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Abstract 


This dissertation presents a collection of protocols for coordinating transactions in a 
distributed information system. The system is modeled as a collection of processes that 
communicate only through message passing. Each process manages some portion of the data 
_ base, and several processes may cooperate in performing a single transaction. 


The thesis presents a model for computation in a distributed information system in 
which the sites and communication links may fail. The effects of such failures on the 
computation are described in the model. The thesis discusses implementation techniques that 
could be used to limit the effects of failures in a real system to those described in the model. 


A hierarchical protocol for coordinating transactions is presented. The accesses to be 
performed during a transaction are pre-analyzed to select the protocols needed to coordinate 
the processes that participate in the implementation of the transaction. This analysis can be 
used to guide the organization of the data base so as to minimize the amount of locking 
required in performing frequent or important transactions. An important aspect of this. 
mechanism is that it allows transactions that cannot accurately be pre-analyzed to be 
performed and correctly synchronized without severely degrading the performance of the 
system in performing more predictable transactions. 


A novel approach to the problem of making updates at several different sites 
atomically is also discussed. This approach is based on the notion of a polyvalue, which is 
used to represent two or more possible values for a single data item. A polyvalue is created 
for an item involved in an update that has been delayed due to a failure. By assigning a 
polyvalue to such an item, that item can be made accessible to subsequent transactions, rather 
than remaining locked until the update can be completed. A polyvalue describes the possible 
values that may be correct for an item, depending on the outcome of transactions that have 
been. interrupted. by failures... Frequently; the.mast sonst: impertant effets of a: transaction (such as 
the payment of money) can be determined without knowing the exact values of the items in 
the data base. A polyvalue for an item that is accessed by such a transaction may be 
sufficient to determine such effects. By using polyvalues, we can guarantee that a data item 
will not be made inaccessible by any failure other than a failure of the site that holds the 
item. 
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A strong motivation for the development of these protocols is the desire that the 
individual sites ef a distributed information system fail . ube 
group of sites be able to continue local processing | ations when a failure has isolated 
them from the rest of the sites. Many of the previous coordination mechanism have only 
(a ee 
Another motivating factor for the development of bea ref el of tr 
applications, the | sae apigetergloacresgh ‘Sy racing the coordinasion 


key words: distributed data bases, synchronization, message passing systems, reliability. 
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Chapter 1 
Introduction | 


Recent developments in electronic technology have sale practical the interconnection 
of a large number of computer systems to form what I will refer to as a distribu ited 
information system. Each of the computer systems (or. sites, as they are more frequently 
called) in the resulting system maintains, some_ information and. tools. for. accessing that 


on. system.may not be. under the 
control of a single administrative authority. A distributed information system allows any 


_ user of any of the individual sites controlled access to the entire, body of information 


managed by the system, while it allows each of the individual computer systems to control 


the use of the tools and information that it holds. 


1.1 Reasons Fer Distrikution . 


There are several good reasons for choosing such an ‘organization for an information 


system. rather than Placing all. of the information. in-a single: larga, shared computing facility. 
I will discuss some. of these.reasons briefly, 


LL Autonomy 


A very important reason for choosing: a distributed. organization for. an information 
system is the autonomy of the. individual sites.. A. recent study (DOH veira 74}: dhais: shown that 
the ability to partition the authority and responsibility for: infermation management. in a 


distributed system is.the most important reason for many. besinesses considering. distsibuted 
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information systems. ina distributed system, each site has control over the information that 
it manages, amd can set its own policies for controlling the availability of that information. 
As we shall see, autenomy thas important implications for the assumptions that can be made 
about the cooperation of individual sites in the execution of processing operations, and for 
the protocols that can be wsed te coordinate such operations. . . 


LL.2 Retiabitity 


A second reason for distribution is reliability. There’ © two ways in which a 
distributed information system can be made more reliable than a-central facility. One way to 
achieve greater reliability in a distributed system is to replicate information, storing it at two 
_ Of more of the sites in a distributed system. Replication increases the availability of 
information in a system with unreliable sites. A single failure does not make replicated 
information imaccessitile. Unfortunately, modifying replicated information is much more 
has gone into the development of protocols to update replicated data, the problem remains 
difficult, and such apidates are costly in that they require ‘extensive communication between 
sites, reducing the economic ib aici at distribution. 


A second source of increased reliability, and ene which I consider to be much more 
important, is the the failure of a single site or communication tik dees not necessarily make 
the entire system fail, while in a single, centralized system, the failure of a single component 
frequently interrupts all processing in progress. The individual sites in a distributed 
information system will ‘be smatier and simpler than a single farge computer system with 
storage and processing power equivatent ‘to the total of that ‘of ‘the individual sites. This 
simplicity should mean that the sites in a distributed system fail tess frequently than the 
single machine of a centralized system. Thus if a distributed system:can be constructed so as 
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to limit the effects of a failure at one site to the interruption of ‘processing that: requires 
information at that site, the reliability of a distributed information system as seen by any 


individual user will be substantially better than that of a'single shared machine. 


1.1.3 Economics 


gts 


A third reason for distribution is an economic advantage that makes a group of 
small computer systems less costly to manufacture than an “equivatent” single large machine. 
A single computer with a certain Processing ¢ rate and  Morage capacity costs substantially more 
than a collection of smaller machines with the same aggregate processing rate and storage 


size. In en to the computing hardware; Lath lacnspaie and software derenervent 


contribute to the cost of a distributed information system. ‘Frequently, the information to be. 


“Managed. can be partitioned in such a way that mast-of the processing operations do not 
require information from more than one of: the partitions.’ Each: partition can be assigned to 
a smal. computer system capable of performing the processing required for the information 

in. that partition. The cost of communication : between ‘sites in such a. systern. would. be 

relatively smail. If the extra cost of developing soliware for x distributed ‘information system 
can be kept smat, @ distributed: information systeny maybe substantially tess costly than an 
equivalent central facility. 


1.1.4 Flexibility 


A fourth reason for distribution is flexibility. Changes in the amount of information 
to be managed by the system can require increasing or decreasing the storage and processing 
Capacity. In a central system, this may require rept 


different capacity. In the distributed system, amg / changes ean frequently be 


ting ¢ the‘entire machine with one of a 
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accomplished by adding or deleting sites, with minimal impact on the sites not being 
changed. | 

Consider, for example, a corporation that has just acquired a subsidiary, and needs to 
modify its adménistrative information management system to manage the new subsidiary. 


Merging the information management systems of the parent company and the subsidiary 


12 The Concurrency Control Problem in a Distributed Information § System 


- Severai ee a ee eT 
system as easy to use as a oe, The subject of this thests,;:and: what 1 believe to be 
. the most difficult of these prcbians, ts controlling the sequencing of user specified processing 


‘Operations. ‘The resuk of performing such processing operations concurrently vhewid be the 
same a s that obtained ty performing them in some sequential order. Before t this problem 


_ Can be discussed in detail, we must have a a more preciee definitian of the way in which stored 


information can be manipulated. For this purpose, I adopt termindlagy that has commonly 
been used in data base systems. 


1 nate csc the exiting iforation apes ofthe past apd the mbndnry may be 
compat, reqring veal no rt fr the. merger Even if the information 


distributed 3 nd this effort should be fess than that t required to merge both into a single 
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The stored information consists of a set of individual data items, each of which 
represents ‘joie independently accessible piece of information. For @ach' data item there is a 
current value that is the information that that item currently contains! A data base state is 
a mapping from the set of items that makes up the data base to. the set of values, specifying 


the current value of each item in the data base. 


The high-level operations that aré to be performed’ on stored infortiation are known 
as transactions. A transaction can be viewed as a function tiiapping one ‘data base state to 
another. Each transaction ts performed as a set of primitive operdtions, caffed accesses, on 
individual data ‘insees Some accesses ti ‘iin itern Cause the dutrerit Value of that item to'be 
changed, and are known as updates. The set of items whose values are changed: by the 
_, transaction are the output items of the transaction.2 The ‘new values produced by the 

' transaction for these items are known ts: ‘the @utput values of the transaction. Each ; 
transaction computes its output values based on the values of the items in the data base state 
that is the input to the transaction. The items that are wed by the the transaction in ‘computing 
‘the output values are refered to as input items and their v values as supplied to _ 


transactions are the i input values of the transaction. 


- The user of a distributed information system views each transaction as a simple, 
complete operation, such as “deposit $50 int account number13642% ‘Each transaction "sess" 
the effects’ of previous transactions in the values that it obtains for its amet ems. A 
problem arises when several transactions are performed seated : Each transaction may 
_see the effects of the:othels on.the shared data dsems:::Inv order to preserve the illusion that a 


transaction is a simple, complete operation, the:trantactions snust be gtomic, in that. each 


1. The term “version” has also been used for what I will refer to as a value 
(Reed78,Stearns76]. 
2. This has also been refered to as the write set of the transaction [Bernstein77]. 
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transaction sees either ail or none of the effects of each other transaction on the data items 
that it accesses. The definition of atomic will be made more precise in a later chapter. 


The probiem of insuring that transactions which are run concurrently are atomic is 
known as concurrency csptrel 2ad is common to both distributed systems and to centralized 
data base systems, where transactions are run concurrently to increase the utilization of 
resources. While there is a great deal of literature on thts general problem, the particular 
control, and make many ef the mechanisms that have been developed to solve this problem 
in centralized data base snamagement systems inappropriate for a distributed information 


system. 


LS Resic Assumptions and Goals 


Two common probtens in evaluating 2 mechanism to solve a complex problem are 
understanding the gusts of that mechanism and knowing the assumptions made about the 
effects of faitares. Tihs section sets forth my own gouls and ausumptions, t allow the reader 
; to evaluate more precisely the mechanism proposed here These assumptions and goals may 
not be appropriate for ail applications, but i.believe that they ace mest appropriate for many 
wees of a distributed information system as-described above. 


1.3.1 Implications of Delay 


A characteristic of distributed information systems. is that communication between 
sites is slower, mwore costly, and jess reliable than: communiation within a site. An 
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implication -of this characteristic is that unnecessary inter-site communication should’ be 


minimized, even if this requires more computation or more storage at each individual site! 


"A second implication of communication detay is:that no‘one site can: readily obtain.a 
view of the global state of all transactions in progress: State information from:remote sites is 
delayed ‘in communication and may be out of date: The tack of: gtebal state information 

makes: concurrency control schemes in which seme decistons:(suck: as deadlock detection and 
backup) are made based on global information awk ward -féer:use: in a distributed: ‘ihformation 
| system. Thus, ideally; the protocols used: for performing trartacions should aliow euch site 
“tO base its-actions on its local state: only, 


A third Implication of delay is that 2 any y operation Anvalving several sites may be 
delayed for a. long period of time before it can be completed. This means ae the 
information should -be organized such that frequent or leaportanit: aperations can be 
, ‘accomplished focally at some site. - While: Iwill uot discuss the: task of partitioning 

information. in detail, I assume that: the operations to be:performed exhibit a high degree of 
“fecality of reference. Each operation requires: only a: small amount-ef the total information 
“available, - ‘and ‘the information can. be aaa that: very few - operations require 


information from two or more sites. 


This assumption is necessary to make a distributed information system practical. It 
seems: quite reasonable for many applications, intleding management. information . systems, 
‘process control,. ond eee compas PAGS gh SES Re aE ETE 


1. I am not addressing the concept of a “multi-microprocessor” distributed system consisting — 
of a large number of small processing and storage elements linked with very pen bandwidth 
communication 


_L3.2 Partial Operebiity 


As noted above, the individual sites in a distributed information system should fail 
_ less often than a single centralized system. ef equivalent processing power..and storage 
capacity. If each site fatlure interrupts only thase transactions wich require resources at the 
failed site, then a tratuaction involving only a anal umber of sites should be less likely to 
be affected by «. failure in a disributed infermatton system: than -would be in a centralized 
system. Ttrus as a goal, the mechani for performing tranmetions should allow a group of 
sites that are functioning aed can communicate with each other te perform transactions focal 
to that group. I refer to this goal as partial eperebdlity, The most_important aspect. of 
partial operability is to allow any transaction that is entirely bocal to one of the sites to be 
performed whenever that site is operating and the request to perorm the transaction can be 


communicated to that site. 


This is a very different form of enhanced reliability fron: that achieved with 
replication, as described by Atsberg et ai. (Alsberg76) 1 believe that the goal ef partial 
operability more accsrately reflects the needs of mon applications: We- shall see later that 
beth. replication of dats within one site and replication of date teemé at several sites fit 
naturally into the mechanism that I am proposing. | 


An inplication of partial operability is that the dependence-of orie site on another to 
perform purety local transactions must be mini¢ised.: - Peetocole requiring. 2 site to receive 
external authorization to perforrh local transactions, ;sudle: ae. that teed by Thomas. itt 
[Thornas76], shoukd be avoided. we ae 


. brought up to date on recovery. 


OR ars ag 


<i. 


A more important implication of partial operability is that error detection and 


recovery are concurrent with the execution of transactions.’ “Backward. error’ recovery 


_ Strategies (Randell78], which stop processing, new transactions. when an error is discovered 


and cause, the data base state to be “rolled. back” to.a previqusly saved, state known to be 
consistent, do not achieve the goal of partial, operability... Because, processing continues 


during error recpvery, a site that encounters an.error can. "get behind” in that it may not be 


Aware of recent transactions. For example, a site holding, a. capy. of a. redundant data base 
may discover that the values that it holds are out of date because, they. do not. reflect 
_ transactions. that were performed. on. other copies. es. The failure recovery mechanism must 


record any information sent to a site during a faiture of that site, so that the site can be 


1.3.3 Autonomy 


As noted above, the autonomy of individual sites in a distributed information system 


is an important reason for choosing such a system over one with a central shared facility. 


"One implication of autonomy consistent with the god! of partial opérabitity ib that individual 


sites should not be depétident on the = ‘as a ‘whole ini’ that they should be capabie-of 


isha 


cation ‘with other’ sites. Thus ‘we cannot 


performing local transactions when not in 
assume that a site which is not in communication with any other sites stops all processing, as 

Wp ara ca ae oe ae, Ce 
is done by SDD-1 [Bernstein77]. 


SYESA 5 Meas aT 


Another implication of autonomy is that each site controls. the Operations that can be 


performed. on. the data stems that it holds. Thus any site may refuse to perform : some 


operation ; at any time. One method of dealing with this possi is to require that each 
transaction. obtain permission ta perform. all of tts component operations Deford any oF ner 
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operations is earried out. This can substantially increase the cost of performing some 
transactions, by increasing the need for locking (see Chapter 4). 


For e transactions, the administrative policies of all of ‘the sites that must 
cooperate are known in advance and examined in determining whether or not a site will 
cooperate in performing a particular transaction. Verifying that a transaction will not 
encounter access restrictions is simitar in principle to verifying that a transaction preserves 
consistency constraints (Le. that it always miaps one consistent state to another). I will assume 
that even though the sites are autonomous, they will cooperate in performing a large class of 
common transactions. Thus in many cases, the acceptability of a transaction to be run can be 
simply verified before it is run, and will not interfere with synchronization. ‘Dynamically 
changing access restrictions must be checked as a transaction is run, and will add to the cost 
of performing and synchronizing transactions. 


1.4 Related Work 


The work of this thesis concentrates in two main areas: concurrency control in data _ 


base systems, and reliability techniques. 1 will discuss the previous, research in these areas 
separately first, and then relate it to this thesis | 


1.4.1 Concurrency Control 


Several papers [Bernatein77,Gray75,Gray77,Stearns76] discuss the problem of 
controfling the concurrent execution of transactions $0 that each sees a consistent version of 
"the data base. Gray et al. (Gray?) give definitions for four different levels of consistency 
and discuss locking strategies to achieve each. Atomic transactions as I have defined them 
maintain the highest fevel of consistency (level 3) defined in that paper. “This is the level 
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that places the greatest constraints on concurrent execution of transactions! The locking 
strategies presented by Gray are efficient, in that they allow the data base to be constructed 


so that a high degree of concurrency may be obtained with little iocking overhead. 


A second. paper by. Gray -[Gray77] discusses a mechanism for concurrency control in a 
distributed system that makes use of the. locking. strategies described in the. first paper. 
While this mechanism performs transactions correctly unless -highly improbable. failures 
occur, it fails to meet two of the goals outlined. above. ..The. locking strategy allows 
| transactions to deadlock, requiring some mechanigm:to detect.deadiock and abort one. of the 
; cnasanions involved in a deadiock in order to allow. the, others. to proceed. . Deadlock 
a detection requires a view of the global. state of .all: transactions in progress, violating. the 


condition of making decisions based on local information. 


The two-phase commit protocol used. by. Gray and others insures that a transaction is 
atomic, no matter what failure occur during its execution. Ifa. failure occurs at the wrong 
time, however, one or more of the sites involved in a transaction may be obligated to hold 

onto locks set by the transaction until the failure is recovered, preventing the execution of 
| transactions local to that site. that set locks which conflict with those set by the transaction 
suspended by the failure. This violates our goal of partial operability. | 


1. While the authors claim that forcing all transactions to see level 0 or level | consistency 
allows transactions to’ be constructed to-sée higher levels of consimency; and may save ‘locking 
overhead by allowing many transactions to run at the lower levels of consistency, they also 
point out that output values produced’ by ‘a ‘tranbaction reflect thé tevel"‘of consistency that 
that transaction. saw. These low-level consistency. y .afe,propagated by any transaction 
that reads them, so that transactions desiring a high level of consistency can never read 
values produced by those observing a-lower level: Thus lowdtevel of a y transactions 
would appear to have very limited use. 
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A study ‘by Dwerns wnd Rosencrantz [Svearns76) discusses a model for distributed 
by a process ‘that ‘migrates among the sites that hold the values that the transaction accesses. 
Each site is Tesponstbte for controtiing the execution of transactions at that site, and the sites 
commuricate ‘arity ~whven 7 ‘revieactton -is ‘movell and shen a trantattion ie-completed. The 
authors describe a diese uf control algoritins ‘that werk ‘by assigning an order to the 
ntrempting ‘to access the same data, possibly by aborting and remarting them. The necessity 
‘of ‘restarting ‘some ‘transaction ‘that ‘tras completed 2 sutsantal amount of processing is 

undesirable, ‘Sat szerm ‘ummvuldable tn this model of ‘concurrency ‘control. Similarly, the 
protocols devéhepell by Grey TGniy77] ateo require Geattiotk Uencton and ‘beckout. 


Several papers [Bernstein 77;Hammer?8 Rothnie71) discuss the SDD-1 database system 
in which ‘the vet of travmsctions ‘to ‘be performed on the date ‘base is-anutyzed to determine 
the amount of tocking needed. Transactions are divide into Classes by the ‘sets of items that 
they read .and ‘write, and transactions in the same class are performed ‘serially ‘with respect to 
each other. Trartsactions in different classes can be performed ‘concurrently. The conflicts 
‘between the sets of items ‘read and written by cifferent classes ure used to select 
synchronization protecdis to ‘be use| to coordinate concutrent transactions from different 


classes. Frequently, transactions can be run concurrently with little synchronization overhead. 


The approach used in SDD-1 of pre-analyzing the set of expected transactions to 
minimize the synchronization overhead for the smoke ‘common. ti -ssactions seems to be very 
promising. The proof that this technique works, (ie. that all trai + ‘tions are atomic), 
however, is ‘so tong and womplicated as to be unconvincing. Making ‘SDD robust in the 
‘event of faitures also appears difficult. The synchronization “protoculs. weed ‘frequently 
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involve waiting for messages that may be delayed by failures. The techniques used to insure 
that delayed messages do not cause excessive ‘delay int the Processing of transactions are 
‘extremely complicated, and may reduce some of the vine of this epmchcttsarion scheme 


by eve additional message exchanges. 


The reliability ai of SDD-1 is also somewhat different from that of this thesis. The 
goal in SDD-I1 is to keep the system as a whole running, even if this means that sites that are 
“separated from the network while involved in a transaction that spans several sites must 


stop. Thus SDD-1 does not achieve our goal of Laie a 
1.4. 2 Reliability 


The work in reliability is perhaps less developed than that on concurrency control. 
An important paper by Johnson and Thomas. Qohgson74) describes an, algorithm for 
updating redundantly stored data such that all copies converge to the same.final value. The 
7 paper uses the notion of a timestamp, which expresses the order in which updates should be 
performed, so that all copies converge to the ‘aes final value, even .if the updates. are 
delayed, duplicated, or arrive out of order. Timestamps have, been used in many, protocols 
for reliable synchronization. This paper does not discuss the. problem of synchronization for 


concurrent updates. 


Thomas (Thomas76] proposed an extension of the ideas in that | paper to provide 
synchronization. An algorithm was developed to allow updates to be performed as long as 
more than half of the sites were functioning. The algorithm is complex, and several flaws 
. were found in the early versions. Another major problem ‘with the Thomas sigorthen is that 


it applies only to cases where the entire data base is stored at each site. 
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Alsberg and Day (Alsberg?6] have developed a robust makti-copy update algorithm 
with a somewhat different approach. They designate one copy as the primary, and insist 
that all accesses occur through the primary copy. The other copies serve only as backups in 
case the primary fails. This strategy eliminates one of the. major advantages of replication of 
. data, that of greater concurrency in access. The algorithm. does, however, seem applicable 


where the only concern is greater reliability, and not greater er 


A. forthcoming paper by Lampson and Sturgis Lampson] presents a general 

_ discussion of performing atomic transactions in a distributed syrem. The paper presents a 

method of storing and updating information in a single machine, such that it is preserved 

and updated correctly even if crashes occur during updates This storage technique is useful . 
for a aia an atomic update within one site. | | 


The last onic i ei papee eves eh cigaritiel te performing updates dt several 
difterent sites atomically. A complicated protocol is used to distribute the updated values to 
each site, such that during most of the procedure, each site carr independently decide to abort | 
the update if messages are siow in arriving. There is still, however, 4 time window in which 
‘a site must wait for the arrival of message from other another site, and cannot decide 
whether or not to abort the update if suct a message is slow in artiving. This algorithm is 
Similar to the two-phase commit protocol described by Gray [Gray77] and that used by Reed 
[Reed78] The Lampson and Sturgis algorithm makes the time window during which a site 
can not abandon a transaction interrupted by a failure quite small by insuring that all of the 
computation done by the transaction will be completed before any site is prevented from 
abandonine the transaction. This is accomplished via extra steps in the protocol and extra 


message exchanges. Chapter 5 discusses commit protocols in much greater detail, 
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Reed {Reed78] is also working in the area of robust synchronization mechanisms. He 
has developed a scheme in which each value assigned to an item can be named as a version 
| of that item. The scheme allows a transaction to obtain.a set of mutually consistent values 
_ for the items that it accesses by. choosing the proper version names This. scheme is subject 
‘to the same limitations as the Stearns and Rosencrantz, scheme, in that a. transaction may 
_need to. be aborted to avoid deadlock. This problem is solyed py having all of the updates 
performed by. a transaction (by creating new. Yersions). be. conditional . until the transaction 

_ has been completed. 


This same mechanism ‘of conditional transactions is used to solve the atomic 
distributed update problem. The mechanism is simple and convincing, but still leaves a time 
_ window in which a failure can cause delay in processing new transactions, — 


1.4.8 Relationship of this Thesis to Previous Work 


This thesis presents a model.for distributed computing that specifies the effects of 
failures on COMPAERLION. . The. model is. similar. to, the. Actors . model..-of computation 
Dtewitr7s, Hewitt77]., The model describes computation, perfacmed by,an,. unreliable system, in 
which components can fail. and failures effect the outcome of the, primitive operations of the 
model. The thesis discusses implementation eee that can be used to insure that the 
actual effec of failures conform to their effects as described ‘in the model. The sein a 

“used build on the work of Lampson and Sturgis {Lampson76] and Gray [Gray77). ~ 


-While much research has been done on.the problems of. synchroniza 
based models af computation. [Atkinson78,Halstend 78, Hewit 7], souch. of this, work has 
centered on developing. primitive synchronization techniques. thas. achieve mutual. excision. 


nin message 
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allow ‘transactions to deailluck, asing same mechentan to detect x deadlock situation and 
abort one of the tranaartions te rewlve the deattioct 2nd allow the others to proceed. The 
mechanism presented an this thenis instead avoids deadlock. “Mats mechanlim allows more 
actual locking of a resource anti! it is needed or soust be locked to avoid debsitock with some 


conflicting ‘transaction which seeds that resource. This approach avoids unnecessary locking . 


that restricts concurrency. 


The aechaniam ane for synchronization in Git thes is tased ‘on control of the 
order in whidh messages ave detivered. Emptomentations of the central algerie that make 
efficient use of the kinds ef communixation aaa a ea ed a 
substantially dees costly than the approach taken by mest of the Work ea synchronization in 
implements speciation ceri wth higher tel pret, 


The technique sted to coordinate transcions involves an anaiyss of the acces 
pattern of transactions that is similar to that used in SDD- DBernstein77], but more fine 
grained in that the actual derivation of each output of a transaction from the inputs to that 
transaction 1s used in ‘She ‘aneitpais, sather than Basing the ansitysts ‘on-the ‘assumption that 
every output of a transaction depends on every dnpet, es #3 dene in-SBD4. This analysis 
Sis be performed with minimal overhead due to the synchronization. _ 
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The thesis includes a proof that itis impossible to solve the “atomic distributed 
update” problem for. all cases in a way that achieves the. goal. of partial operability, given the 
semantics of the model presented here. The proof applies. sacl advanced »Y nee 


Bhd +hwins 


A novel approach’ to the atomic distributed sae problem is presented. This 
_ approach involves keeping’ several ‘current values for some data ‘items, and builds on the 
version naming synchronization schemes. of Reed [Reed78) and ‘Stearns et al. [Stearns76] 
This. approach is not limited to the particular synchronization scheme discussed in this 


hemes discussed above. 


thesis, but is applicable.ta any of the synchronization sche 
To surhmarize, I feel that the actuals aaa of this thesis are: 


A model for distributed. computing in which. the. effects of failures 
are well specified and wi cenapoenes fechaiques: for eens these 


\ specifications — a egg t 


A techniqare “for coordinating what I refer to as-an “atomic 
‘braadcast” that can be implemented efficiently in the kinds of 

computer networks ‘currently ‘used to confiect ‘sites in distributed 
information systems _ 2 ; 


A technique for analyzing a set of transactions to be, performed to 
determine which ones can be performed without locking 


. A mechanism for ‘locking data items’ at’ several sites in order to 
_ perform a distributed atomic update without allowing a care’ to 
‘delay actess to the locked data mshin in most cases 
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1.5 Thesis Pian 


wutitig that is used 

site that behaves as 
Hhiidinihiy thiiishctions is 
formulated in terms df this inodél. The chapter discussés sevetdl, ways ih which the order of 


throughout thé thedil. 


specified by thé prbchls iiddél de distiited. The probiein UF fy 


execution of trinédctions cin be coritrolied, and stows that ofily one of these achieves the 
goai of partidl operability. | 


 Chaptet 4 idetiniss ad siinple synehtbiiiatio prObiéth (RAL Gdhsined UF edoraitiating 
edi, Ati Sibtiie brdidiciie GistHiButés a abt OF iridssdged to 
a set of receivers stich that the order in which any one receiver sees messages from several 


what I refer to a8 dn atgiiie Bike 


such broadcasts is consistent with the order i which the broadeaits are received by any 
other receiver. A sifipie iniéclidhiiihi 06 petfotii: ‘hie tisk presented.” ‘This mechanism 
forms the basis of the syhietitonization rnechariliri) for eohclirréit. transactions discussed in 
Chapter 4. linpleriientations of this mechanism that take ddvaritige of the synchronization 
constraints imposed by the cominiinication network a are discussed. | These iinplernentations 
distribute the miessages with very little overhead aniblialle te tHe ‘enforcement of — 


synchronization COMStHAihts. 


eee os discusses a Lance: of — tranaetions A n cebtes for 
‘discussed. THis ahalydis is used to show that correct sieht of all transactions 
cannot be accomplished with a protocol that achieves the goal of partial operability. Three — 
different clasées Of transactions are distinguished, on the basis of their access patterns. A 
mechanism that builds oti the atomic broadcait itiechanitm of Chapter 3 is presented to 


perform trangactions. This mechaiiism can be tailored to minimize the cost of synchronizing 
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transactions that are expected to be performed frequently. The mechanism is general, 
however, in that any transaction, expected or unexpected, will be correctly synchronized. 
Unexpected transactions have little impact on the efficient operation of the synchronization 


mechanism for the expected transactions. 


Chapter 5 considers the implications of the need for locking on the goals of partial! 
operability and autonomy. These goals dictate that a site that has set a lock for some 
transaction should be able to decide to abort that transaction if a failure interferes with the 
prompt completion of the transaction or if the transaction violates the access policy of the 
site. I show that there is no protocol that can be used to insure that no failure can prevent a 


. functioning site from promptly completing or aborting a transaction requiring locking. 


Asa solution to this ‘problem, I propose a novel mechanism that allows locked data 
‘items | to be made available to other transactions before the completion or abortion of the 
locking transaction is decided. This. sche is appropriate for systems in which the 
ability to perform transactions in real time, without long delays waiting for locks to be 

“released, is important. | 


. Chapter 6 presents a comprehensive example showing how to apply the techniques of 
this thesis to a typical distributed information system. The example is an inventory control 
system described in a report on SDD-1 [Bernstein77]. The techniques of this thesis are used 
.. to develop a robust synchronization scheme for this example with little overhead due to the 


synchronization. 


Chapter 7 summarizes the new ideas in the thesis and discusses areas for future 


research. 
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Chapter 2 
The Process Model of Distributed Computing . 


. ‘This chapter present the model for distributed computing that will be used in 
discussing synchronization ina distributed information aaah ane first section presents ¢ the 
model which includes seiacifications of the effects of failures on computation expressed in the 
model. Implementation strategies for ‘limiting: the: ak geal Of netiaal oe to the to the 
failure effects specified in the model are discussed. The second section poses the problem of 
performing transactions (as described in Chapter Ny atomically in the framework of the 
model. Various techniques that could be used for Syocnreniana, are discussed to show that 

only ‘one of these can be used by a syatem that chief the geet of partink operability. 


- 2.1 The Model 


eat Based on the assumptions and goals set forth ip the previous chapter, I wil now 

describe a model for compation a" a cist eared information System. In order to. centralize 
. the description, this chapter presents all of the model, even though some of the concepts will 
not be used until much later in the thesis. This model includes ube forms of communication: 
message passing, and changes in state observable by later computations. Bissige passing 
may occur between sites or within one site. Communication through state changes, however, 


occurs only within a single site. 


2.1.1 Definitions: 


The basic unit of the model is a process! A process can be. viewed as the unit 
within which communication through state changes can occur. A process consists of a local 
state, a set of iDput poets, and a se of process step specifications The computation 
performed by a process takes place in a series of process steps. A process step maps an 
“Input local state and a set of input messages into an output focal state and a set of output 
messages. Each process step specification specifies the form of a process step, by stating: 


A set of ipput ports for the step.. late gacmeias 
step from each port in this set. 


The outp foal sate as a function of the Inpot local state and sed 


A set of output messages and their destination ports. Both the 


message conteets.and xs ports. be spedfied. as. 
fcr fhe Spat el wate and he mesg eed 


An important point to note about a process step is that it computes its output 
messages and output local state. EGS BAER proces eae CAR. Se Mees: opel 
‘computation on the local state of the process and the messages received, rather than simply 
retrieving information. from the local state or storing information in the local ‘state in 
response to messages. This capabitity of «process sep to perform computation is used in the 
implementation of a transaction as wil be dscusted in Chapter 4 


1. The word process has been used to denote a number of ilf specified concepts in the 
literature. My use of the term process is not inconsistent with the common usage of the term, 
however the reader should realize that the term has a very specific meaning in this thesis. 
Other terms that have been used for very similar concepts are Actor [Hewitt?6], and message 
handler {Reed78]}. 
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Conceptually, each process resides at one site, its home site. The home site of a | 


process is the location of the process state of a @rocess. ‘Fhe tiome, tite also is responsible for 
carrying out process steps. The fact that each process is implemented at a single site will be 


used in determining the effects of snllures on the execution of process steps in this model. 


_ Each of the process steps ofa process is atomic with respect to the other steps of that 
process. The output local state of one process step becomes the input local state of the next 
step in the sequence. The execution history of a process consists of the sequence of: steps 
that have been performed by that process. For each process , there is an ordering <p on 
the steps of p, such that s, <p 89 if 8; preceded S» in the execution history of p. 


|The set of messages that a process has received in each ‘of its process steps and the 
. initial local state of the Process form a ‘complete description of its execution history. From 
the flats Js received at each sep and the process step specifications, one can deduce the 
. messages that are produced and the changes made to the process sate. The input messages 
‘to each step can be represented by a set I of [message,port). pairs describing the messages 
. received and the ports at which they were received. 


Figure 2.1 shows an 1 example of an execution history The ‘igure shows a list 
describing the input messages to the process mae of P. The frst process step of P ‘is 
represented Py the bottom entry in the is, with subsequent process seps higher in the list. 
This list may be thought of as a log that records the messages received by P. ‘When a 
process receives messages.at a single port only, the execution history can be reoresanted bya 


list of messages received, as each step receives a single message at that port. 


|The foal sate ofa proces private to that proce and can only be changed by 
_ (process steps of that process. Stenlary, for each por, there is a single process that receives 
the messages sent to that port! The tack of “sharing” of ports or tocal state between 
| rocetes greatly smite the mpementton of precee in it tbuted ape. | 


The characteristics of a process stated above (sequential execution of procew steps, 
each process at one home site, and no sharing of process sate) describe the behester that 
process must exhibit, not és tmplementation. in practice any implementation that behaves 2s . 
described above is satisfactory. In an implementation of processes, for example, process steps 
may be execited concerrenty 0 Jong a the behavior is the mame as that produced by 
sequential execution. 


1. Note, however, that several processes may send messages to the same port... 
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One can view the execution of a process as being performed by an ‘interpreter that 
carries out the execution of all of the processes in ‘a system. This interpreter maintains a 
local state for each process and a set of messages for each input Port. One cycle of this 
interpreter selects a process, step specification of some process, selects a message. from each, of | 
the input ports for that step, and carries out the selected step. The interpreter deletes the 
received messages from, the sets of messages for | the Input ports, changes the local state of the - 
proves: and adds any output messages produced to the 28 of pending messages | for the 
| ; appropriate ports. . 


The interpretation can be distributed . (one imerpreter for each process) because the 
| only. interaction between steps of one process and steps of some other proces is the sending | 
of output messages produced by process to inpu 
| inercion can ey be scomplied by, menage pug bee the dub 
i : iresar provers, 


t ports of some other process. This 


: 212 Effects of Failures in the Model 


: The process Hep specifications completely specificy any comeraton ‘taking pace in 

| the absence of failures of the underlying mechanism that carries ‘out the | Process steps. This 
section discusses the kinds of failures that can arise in a diserttvated information system and ; 

their effect on the execution of process steps and on message passing between processes. ‘Two 

"extensions of the process model to include a specification of the. — of failures on 


"age computation are © presented. 


2424 Types of Fatinres 


Two different kinds of futures can cocur in a distributed information syiee: she 
 sinaies, ad Geneualenaea ‘thou. A se ature is lilt to some se, and can cause 
_ processing at that se 10 be sunpanded, or cause wren fated ti that ite t0 be tost oF 
‘damaged. A communication failure causes inter-frocies manages to be let, delayed, 


"damaged, or delivered to the wrong recipient. ee een oe 
_ to hide the atic of taerts Tom 8 user sompioness | 


Aa error detecting code, or checksum, canbe used to detect messages that have been | 
" damaged of delivered to the wrong recipient ‘White it ts iophisibte to detect aff such errors! 
the probability of sndetected comminidilion errors cin ‘be’: > de arbitrarily small by 
_ increasing the proportion of aach reseage Seveted ib error dalection. 1 will therefore make 
arbitrarily small, but nen-zero probably of an undetected error. If any message that is 
found 10 be in error ts discarded, the pombe etic of communication errors are United to | 
eet eemee 


“The mos commen sffe of a ute flluce i that computation at that ste i 
- tempararily suspended, apd that some of the | as areata Jn progress at the time of the 
failure may be fost. If ate atte were to cor daring the execution of a process sep, thet 


a Noieune ad alo Gok aoe eee a 
communication faihwe will cause a meengs to be transformed sbahen tt appenss debe aerrect 
to the error detection mechaniam fut does not correspend to the original message. 

2. Many communication syitems exhibit anather fathure mode in which a message is 
duplicated. in designing a communication protocol, one has a choice as to whether to 
guarantee that ail memages are delivered reliably, possibly delivering some twice, or to 
guarantee that each menage arrives at mast once, and thet some messages may be fost. | 
have chosen the latter akernative. heap cans popes praca paras. Agana 
ROmoecad: Comenmenlcatiems wenn Sieben te altacts af Nath of Sanus Since of Selbace: 
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step might be left partially completed, with the focal state of the process corresponding to 
_ neither the input state nor the output state of that step. This ‘can be prevented by using a 
* robust storage management technique for storing thé focal state of a process. Such a 
technique aflows a group of updates to be made atomicaitly to iiifurmation stored at one site, 
such that if a failure occurs either‘all or none of the tipdates uke place.’ ‘T'he atomic’ stable 
storage mechanism of Lampson and Sturgis [Lampson76] is such a technique. A description 
of all of the upeates to be performed, known as an intentions 5 list, is formed and written to 
permanent Storage in a single operation before any of the updates are carried out. A failure — 
occuring before the intentions tist is generated or. one inpertertng with the writting. out of the 
_ intentions list causes none of the updates to be performed. Once the ittentions list has-been 
written, however, the error recovery mechanisny caft Use it 'to insure that all‘ of the updates 
specified will be made, even if the site making the updates fails after having partially 
completed them. The write-ahesd-log protocol of Gray {Griy7M lso provides the same 
Capability ‘for making a collection of updates atomically; by writtig out a description of the 
updates to be made to a log tape before any of the updates are made, 


Each process step can be Implemented as an atomic update to Saye storage. ‘This 
implementation insures that a site failure leaves the local state of | a process executing a 
process state either af the input state to that step ofthe output’ sate of that atep, and not 


some: intermediate state or mixture of the two. 
2.1.2.2 Two Ways to Include Failure Effects in the Process Model 


By using the low fevel implementation techniques discussed above, one can constrain 
the way in which faitures affect execution of processes. ‘By atgmentirig the definitions of the 
Process modet to include specifications of the effects of failures, we can produce a modet that 
“describes computations in a “real” distributed information system in’ which site and 
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communication failures can occur. The. choice of the specifications of the. effects of failures 
should be made so as to reduce the effects of actual failures on the model, but also to be sure 
that an implementation of processes in which the effects of failures. are limited to the 
specifications can be obtained. I consider two different faiture specifications, one that is easy 
r implement and one that is harder to implement but limits the effects of failures more 
severely. | 
2.1.2.2.1 Simple Processes 
Using basically the techniques described above, one can build an implementation of 
processes in which the effects of a site or communication. failure are limited to lost or delayed 
messages. This is done by using error detecting codes. to detect communication errors. and 
discard messages that are in error, and storing the local. state of each process in. atomic stable 
storage. Some care must be taken. in the implementation of a process step te.insure that no 
possible failure causes memages to be apparently. duplicated, by causing some part of a 
process step to be repeated. If a process step is. restarted, after: being partially completed, 
then it may send out the same message twice (once before being restarted and once after), or | 
_ may modify its focal state as if it had received the same message twice. 


These undesirable effects can be avoided by performing a process sep in three 
stages. First, delete any record of the input messages.to the step so that.a site failure 
occuring at this point would cause them to be lost.. Then, perform the process step and 
update the local state of the process to reflect its completion. Finally, distribute the output 
messages of the process step to their destination ports. A site failure occuring before the local 
state of the process is updated can result in the process step not. being performed, or an 
apparent loss of all of the input messages to that step. A failure after this point may cause 
output messages of the step to be lest No failure causes the local state of a process to be 
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process step to appear to be duplicated. 


A less likely result of a site failure is that the information stored at a site in 
permanent stable storage is damaged. This can be detected, with high probability, through 
the use of error detecting codes. As with communications faihires, however, it is impossible 

to detect all such errors. The local state of a process can bé replicated within one site to 
dectesise the probability that a fatture will destroy all copies. A protess step-is implemented 
aS an atomic update to all of the topies of the process state. Ariy cdpy of the local state of a 
process that survives a site failure can thus be usetl to becoiié thé Current local ‘state Of the 


process. 


To summarize, the effects of a site failure can be limited to lost messages (through a 
process step that was aborted after receiving messages), or delay Of processes at that site. 
This is achieved by using atomic stable storage to ‘represent ‘the local state of processes, 
replicating local states, using error an error detecting todeé to detect damage to a local state, 
and indefinitely suspending any process for which no valid local state can be found. 


Limiting the effects of failures: to fost messages or delayed execution can easily be 
achieved without excessive communication or processing. overhead. Many applications 
require a higher degree of reliability. In the next section, I discuss a different 
implementation of processes that gives a greater degree of reliability with greater overhead. 


2.1.2.2.2 Robust Sequences Precesses 


The effects of failures on simple processes are well specified, but still undesirable for 
_Most applications. For many applications, guaranteed delivery of all messages sent by a 
process to a port is desirable. This isa very difficuk constraint to express, as indefinite delay 
/_ Of the delivery of any, particular message by a communication failure cannot be prevented. 
In order to clarity what 1 mean by guaranteed delivery, | will-introduce a constraint that I 
_Tefer to as sequencing on the delivery of messages. Sequencing implies that messages sent 
from ome process » to a port q are received at q in the sarue order in which they were. sent 
| by pi Robust sequencing ieplles in asiition that no messages ate let 


_ For each port q define the ordering <q on the messages received at port q to be the 
total order in which these messages were received. Fer qach.peoress » the ordering <» ov 
q, the order <, in which the messages sent by p are received at.q is exactly the same as the 
enter in which the steps that produced those messages are ordered by <p This. means not 
only that the messages are received at q in the same order in which they were produced, but 
also that there are no gaps in the sequence of messages received. Reception of message m 
SOO 2 ae cl ae repent nor memege i See m. | | 


Schoen acess “Wa. ghee “sme: ol anscacion OE 
process steps from the actual communication of messages from one site to another. This can . 


be done by maintaining a process database for each process. The process database for a 


process » contains the locat state of , an input message queve for each input. port to p, and 
an output message queue for each port to which » has sent a message. Each output 
message queue contains a list of messages and a transmit sequence number (TSN). The 


-39- 


input message queue for a port'q contains a list of messages ahd’a set of receive sequence 


~ Mumbers (RSNs), one for each process that fas seft’a:hesiagé'to q!’ A-process database is 


stored Using. atomic: stable storage, so that a site failure during modification does not catise a 


ro database to be left in some intermediate State. 


A process: step of p can now be implemented ‘as ‘an atomic: update to the process 
database of. p, which removes the messages ‘received: by thet sepfrom the input message 
queues, changes the local. state of p, and appends the: messazes' produced By ttritt stép to'the 


_ output message-queues. (If there is:no queue for some destination port, a new one is created). 


Maghlw wees? 


Messages can be transfered from an output message queue of a process p for a 


destination port q to the input message queue for q with a robust communication protocol 


using the sequence numbers RSN and TSN. Briefly, each ‘site! periddésailly attempts-to send 


the first message in +) non-empty seat ih queue, anne wes TSN of met queue to the 


message sent. When the site holding port q receives a menage sent from p. it verifies that 


the ‘sequence number attached to that message is equal to the RSN of q for p, and it so 


updates the process database of the process associated we port q to add the message 


received to the end of the input queue for-q, and to increment the RSN of port q for p. 


Whether or not the. sequence number: of the message received is‘correct, the receiving: site 


“sends.an- acknowledgement ta: the site holding. p containing the RSN of q for ». This 


_ acknowledgement. informs the sender of the most: recently . received: ‘message. ‘The 


acknowledgement. either acknowledges: receipt-of a menage Es informs. the: sender that 


retransmission of some message may be required. When the site holding p receives such an 
acknowledgement, it verifies that the sequence qusaber: in: soniemraritas. nites is the same 


as the TSN of the message queue for q in the process database of. frand. if 50 deletes the . 


first message in that queue and increments the TSN. 


ae 
7 I will not at Chis point explain how the message queves are initially set up when two 
processes first begin te communicate with each other. This is somewhat complicated and will 
be discussed at lungtle i Chapter 3, where a use for rebust sequenved: procemes is. discusset. 


This implementation of processes guarantees delivery of inter-process messages in 


sequence. The cust. of. the-protocol is-the extra messages (acknowledgements) used, and the 
storage required for the memage ques: and:sepenee nerbers: This cost is smatt if each 
| process converses with selatvely few processes, and if mesmges inv thd output queues are 
promptly forwarded.! Inv the synchronization protocols sse@ ii this: thesis; each’ process 
converses b directly with alta ee communication 


is small. 
2.1.3 A. Justification for This Model 


A numberof models have been propote for ditibuted computing. I feet that the 


model described shove best reflects the goals and suumptions ofthe Kind of distributed 


information system discussed in Chapter 1. 


One semantic model thet has been proposed for disriborad computing is the object 
model [Liskov?7,Sahtzer78i In the: object modes, Informaton ts reprecnted by typed objects. 


For each type of ottject,. there. it: a set of: operations, sith: as ade . sabtrace, multiply, and 


divide for integer objects, wiricl car be used: to: manipulate objets ‘of that type. Ter sdainen 


A Any eg ee ee ee 
refuse to execute a process step for a process with non-empty output queyes, and can refu 

to acknowledge a message dent to: ¢ putt Witlk # dh “Tivese Nese diecadies do, 
Se ee plication: re . buffering. betweer 
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describing the operations that can be performed on ‘abies of the new type and a 
representation for objects of the new type. Canipuaiic is Aa as Scum ‘of 
operations o on the sets Hebi , - > a: 


While the a abe model is a very natural one for many | users, several ree arise 
in the application of the object model to distributed computing, The most. serious of these 
: problems is that it is unclear what the appropriate semantics for accessing remotely managed 
objects: should be. Many suggestions have been made, including treating an Object references 
uniformly, whether local or remote, treating references fo remote ‘abjects “specially ‘and 
"maintaining a local copy of the remote object; ‘and Wisallowing ‘references to remote objects, 
aind instead” using “meisage oriented communication” between’ sites. ‘The first of these: 
suggestions is difficult to beiadesuaals while the othefs violate ‘the conceptual si ia of the 


bas 


“object model. 


The uniform object model (in which a user computation does not distinguish 
between references to local and to remote Objects) is difficult to’ implement reliably. 
Operations that involve ‘objects at different sites can fail ‘indifferent ways (due to the 
possibility of communication failures) than operations ‘on objects ail at one site Hiding the 
“different failure modes from the user is difficult or ‘rnpossible, forcing the: user to deal with 
the problem of determining what the outcome of a ‘aquence of  Specions on a will be 


if failures interfere with their norma! sail ale 


Several similar semantic models based. on. message passing: have been developed for 
_ distributed computing. These include Actors (Hewitt76], the y-calculus [Halstead78], and 
data flow [Dennis75) These models in their pure form all describe computation such that 
the only communication between primitive computation events is through explicit message 


passing. 
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The Actors model provides a uniform way of describing any computational activity 
as a group of events each event being the reception of as message by an Actor. One 
problem with the Actors. modet is that exactly which Actors are primitive, raplementing their 
effects directly rather than sending messages to other Actors to achieve ther effects, is left 
unspecified. Thos i sometimes swt to det with the Adrs tod, ae there i atways 
another levet of description “below” oy Se ek ee Sone . | 


Side effects are introduced into the model with the notion of a cell an Actor that 
“remembers ne of the menage: that it has been sent. snd repent that mesenge on request. 
This primitive mechanisy cart be used for modeling. computation in. which the processing, to 
_be-applied to some message is not known in advance and is dependent on some future event, 
such as storing a data item for later transactions. Cells are iso used to implement events in 
| which two or more messages are logically “received” (by using cells to store messages), as.the 
Actors model does not allow an: Actor to receive two or more messages in 2 single event. 


The y-caicutus is similar in principle to the actors model. It, however, provides a 
mechanism for introducing. primitive functions that are not implemented by message. passing. 
Cells are used in this: model as well, for storing values for later use. In addition, a 
“mechanism called. 2 teliep: is. introduced to provide » way for a pair of messages to be 
of a distributed 


received iy One. event. White the token mechanism is more. : 


cell, it is stilt rather hard to wrdderstard. Moreover, the im 


oy oe 


system based: onthe eae mae ek din at he protess preted for one 


such en eer re ring 
tokens. 


’ celts and 
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Data flow schemas have frequently been used as a tool for describing repetitive 
processing, such as computing a Fourier transform. A data flow schema provides a natural 
mechanism for events in which two or more mesiages are received, unlike the above models. 
Unfortunately, computations in which the processing to be applied to some, message depends 
highly on the contents of the message are hard to describe in data flow. Recursion and 
iteration are somewhat difficult to express naturally, and greatly add to the difficulty of 
implementation. A data flow description of computation where a lot of information is stored 


for later (unknown) use, such as a data management system, is awkward. 


" The process model previously described is an attempt to bring together. some of the 
good: features of the models described above, without the ‘disadvantages. The two different 
forms of ‘communication provided in the process model represent the properties of 
. communication in a distributed system better than either observation of state changes or 
; message passing alone. “It is easy to specify the effects of a failure in a system based on 
processes, and to build an implementation of processes that meets. the specifications. 
Distinguishing between intra-process and inter-process communication encourages the user to 
plan his application carefully so as to minimize unnecessary communication between sites, 


_and to plan for site or communications failures. 


The priceds. model also captures the concept of autonomy. . All stored: data’ is 
represented by the locat states of the processes. No process canbe “coerced” into performing 
some function for any other process. Alt access’ to stored infitinattoh ts mediated by some 
Process that can implement its own access control policy. This allows the problem of access 
_ control to be largely igncred in the model, as ach process can provide its. own access control 
policy. At the same time, the process step specifications of a process sperity the access control 


policy of that process by stating what the process does in response to the messages that it 
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receives. “Tvus 2a ‘eer iimplententing ‘some ‘application can ‘examine ‘the .process ‘step 
specifications of precesses providing ‘services that ‘tre wishes ‘to use and cain in Muiny cases 
determirre «whether ‘or wot access ‘restrictions will: be encountered :in his application. 


Pilso Aritrerent in ithe (pracess ‘model is ‘the notion ‘that ‘some processing activities can 
tbe performeli ‘simply ‘by -une ‘site. ‘Although ‘each process ‘has ah ‘overall specification of its 
operation, ina resl:gystem:mott processes ‘will'be iniplemented from smatier pieces. I will not 
specify winit ‘those :pteces ‘are, us ‘the implementation ‘of 2 [process could ‘be ‘based ona 
depending on what is deenwd ‘most convenient. ‘Within one process, however, one need not 
deal with ‘the special problems ‘of a distributed information .aystem, as ‘each process is 
executed soldly at one site. | 


‘The mechanism ‘used ‘to specify a processing ‘event ‘that logically receives messages 
from two ‘or ‘more ‘sources (multiple sports) .seems ‘much more ‘natural .in the process model 
‘than the mechanisms ‘using ‘célls ‘or ‘tokens. ‘As events in whith two or more messages are 
received ‘are ‘common in many-applications‘and ‘can ‘be‘construceed from ‘the primitives in the 
‘Actors or -cailetlus ‘moddls, ‘there ‘seems ‘to be no reason ‘not to: inclutie this Important special 
‘case in the model. ‘Inclusion ‘of ‘this capability : identifies ‘for ‘the: ‘implementor of the system 
the cases where two messages ‘ure being received by what is. lagically one process step. This 
‘makes it simpler ‘to ‘Consteuct aan ‘efficient and -tobust ‘implementation than Af the multi-port 
Teceive were ‘simulated using some more.general-mechanism. 


‘The association ‘of several independently named ports with ‘one process is a very 
useful feature of the ‘process model. It-can be used to. group several: independent. as 
activities that wish to communicate via a shared data base in a single process. ‘Such 


processing activities can be implemented as independent process step spécifications of the 
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same process, each of which receives its input messages throug h a different set of input ports. 


This use of processes is similar to a monitor [Hoare74] or a critical section. . 


A second, more important feature of ports is that they provide a way to classify the 
messages sent to a process before messages are received. One application ‘of this ‘capability 
would be a process with several queues of pending messages that are serviced with some 
| priority algorithm, not necessarily in the order in which the messages arrived. Ports also | 
allow a process to temporarily ignore one class of messages while exchanging messages with 
other processes to complete some processing’ activity. “TWiis use‘of ports will be demonitrated | 
by the locking aratngy discussed in Chapter 4. _ 


The differences between my model and the others are'a reflection of different goals. 
. My ‘model is an attempt to provide a way to express ‘applications for a * distributed 
information ‘systém clearly, such that the effects’ of failures ale’ well ‘Specified. Others have’ 


2 ER WD 


_been more concerned with formality and minimization of the Lob sia ci 


2.2 Atomic Transactions Revisited 


This section examines the problem of performing transactions atomically as expressed 

in the framework of the process model. We show how the simple definitions of transactions 
given in Chapter ! can be stated in terms of the process model and show how to express the 
property that transactions are performed atomically as constraints on n the order of execution 
of process steps. Several mechanisms that could be pe to control this order of e execution to 


achieve atomic transactions are discussed. 
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. 2.2.1 Expressing Transactions in the Process Model 


Recall that a transaction is a set of accesses to stored data items. The definitions of a 
data management system given in the previous chapter can be mapped onto the process 
model by using several processes (at least one for each site) which I refer to as dats 
managers. Each data manager maintains some of the data items as components of its tocal 
. process state. The process steps of a data manager perform the accesses te the data items 
held by that manager. If a data item is replicated, with several sites having copies, then 
_ several data manager processes maintain copies of that item. | | 


A transaction in ihe -peoceds model consists of a set of process steps of the data 
manager processes which together carry out the accesses needed to perform the transaction. 
Each data manager may perform several steps in carrying out a single transaction. If 
communication between managers is required. to perform a. transaction, . then the. output 
messages of some of the processes steps that, perform that transaction: will be used as input 
messages in some of the other process steps performing the same transaction. 


In addition to the data manager processes, which implement accesses to data items, 
_ there are transaction processes, which perform the function of transtating | from a high level 
description of the transaction to a set of messages to be sent to the data managers. These 
Messages direct the data managers to perform the necessary accesses to carry out the 
transactions. A more detailed description of the function of the data managers and 
transaction processes is agivan in Chapter 4, which discusses mechaniuns for performing 


tran sactions. 
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“2.9.2 Performing Transactions Atomically 


. Intuitively, a transaction is atomic. if.either. all or none.of, its. effects are visible to 
», other transactions, There.are two ways in which one transaction. may. observe the effects of 

other transactions: messages sent by. steps. of one transacdon that are received by steps. of 
another transaction, and the. modifications of local process: states, hat are, made.by steps of 


one transaction and later observed by steps of another transaction. 


The -first method of observation, . direct message passing, rarely -occurs. This is 
because a transaction is.a.complete, independent processing activity and, dges not in general 
communicate directly with other transactions. The exception. tp. this case. is. that the user. who 
_Submits a transaction, (by sending a message nto, the distributed information systern) may 
_ know of other transactions by having received, messages. sent from. other transactions. 
Controlling _ Sequencing of transactions. so that. the order of transactions as perceived fram 
explicit Message’ passing is consistent with their order as perceived from. observations of 
modifications to focal state is relatively simple. For the moment, I wil] presant.a. definition, of 
atomic transactions that ignores this method of ae aces a 3 discusses this 


| problem further. 


- The second source of communication between transactions, the local process states, is 
much more important in most applications. Recaff ‘that ‘for ‘each process p there is an 
ordering relationship <p! that defines the relative order of occurrence of Process steps of p. 
These local ordering relationships can be used to define an. ordering of the transactions as 
follows: a. ‘ | 


Transaction T, < To iff there is a process p and process steps 5; and 
89 of p such that 5; is a part of T),and'sy isa part of T, afid 5; <p 
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Thus two transactions are ordered if both contain steps of the same process. This ordering 
is a reflection of which transactions may have directly observed effects of which other 

“transactions. A transaction Ty can also observe the effetes of suttie transaction T's indirectly 
if there is some transaction Ty such that Ts < Ty (because of the order of process steps of 
some process p) and ‘T'y < Tj (eeceuse of the ofdiét of pitdess seps'of some’ other process ¢. 

~ Indirect observation can cccat sc riot reine a ce al the Viitiies 


| that that transaction saw. 


| The condition that we require for a transacttdn to be stomi Is that either all or none 

of its effects be reflected’ in the data -valuey seen by’ oiler triimmctions “If a group ‘of 
transactions is performed: atornically, thén the effects ‘of those “transactidné (modifications 
made to the values of dath tems and meiiagés produced by by thW thinsiidtions) are the taime’as 
“if, the transaetions were performed serially in’ soml ‘sequenice:'Wit- exch transaction ‘being 
- entirely conmpleted before the text transaction ii thé sequerice Wi Béguin. “This: requirement 
can be. expressed ie ne Cease ee eee 
transactions as follows: | : Ss Sere oa 


Transaction t is atomic with respect to a set of transactions. T if 

_ there is no sequence of transactions t;, .., t,, in T such that t; < t;,) 
| fori gi<n,andt, <t<t, Equivalently, a set of transactions is ; 
atomic if the transitive closuré! of tte < tering, fii a ames 
order on that. set of transactions. 7 


In order to insure that a set of transactions is performed atonal, we must insure 


that the < ordering resulting from any concurrent execution of those transactions is cyte free. 


One way to insure this is by choosing the assignment of data ttems to data managers ‘such 


f 
5 Throughout this thesis, I. will: use: the ee ®. cinemas a reflexive transitive 
closure (i.e. x <" x for any x). 
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‘that for each transaction there is a single data manager that can perform that transaction. 
* Thus each transaction is seen by only one data :manager,.and.ne cycles in. the < ordering can 


-- arise. 


_ This approach can be rejected because the suignment of data items to managers is 
an not ‘solely under control of the system designer. The autonomy of individual sites dictates 
| that certain items must be managed by processes at ‘certain ‘ites. ‘Some transactions may need 
to access data items from several different sites. Because each process must be executed at 


‘one. site, there: is: no way to have ane: dara manager pres perform a.transaction at several 
sites.” 


. Perhaps a more serious objection to 0 this Proposal is that it makes the addition of new 
transactions, which access items in patterns that were not planned, -aumicakt or linpositbte 
Adding anew transaction may require complete redesign of the syaem so as to allow new 
transaction to be performed by a single process. | a 


| We therefore must show ‘how to coordinate transactions that involve process steps 

from. several different processes. This c can be accomplished by controfing the order in which 
the data managers perform the process steps which perform accesses of transactions. The 
- next section discusses four primitive mechanisms that ‘could be used in ‘coordinating, the 


MPR MS Sitaes 


process ape of the data managers. 


1. Recall that the specification 0 of the effects of failures on the execution of a a process was 
greatly simplified by the fact that each process is executed ‘at obe site: ‘Therefore, we do not 
wish to abandon this assumption. 
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2.2.3 Primitive Synchronization Mechanisms in the Process Model 


There are several mechanisms available in the process model that could be used to 
cofistrain the order in which processing operations are performed by processes. These 
. mechanisms could be used to construct a solution to the problem of performing transactions 
atomically, in much the same way as mechanisms such as Semaphores [Dijistrast] or 
| Monitors (Hoare?4] are frequently used to consruct sotutions 0 other synchronization 
“problems. 


To achieve the goal of partial operability, the synchronization scheme for 
transactions must allow a transaction that is purely focal to one data manager to be 
performed whenever a request to perform that transaction is vent to the data manager. Thus 
synchronization mechanisms that do not allow such transactions to be performed promptly 
should be avoided. The goai of partial operability will thus serve a acm in demic! 
synchronization techniques for transactions. . 


One synchronization technique that has already been introduced is the sequencing of 
messages sent between processes. Sequencing consists of guaranteeing that messages sent 


from one process to a port are received at that port in the same order in which they were . 


produced by the process. As we shall see in the next chapter, robust and sequenced message 
communication is sufficient to ica proper synchronization of many kinds sa transactions. 


Sequencing alone does not compromise the goal-of partial operability. The only case 
in which the constraint of sequencing prevents a message sent from a process p to a port q 
from being promptly received and acted upon is the case in that there is a previous message 
from p to q that has not yet been received at q. Using the implementation of robust 
sequenced processes described earlier in this chapter, this situation is quickly” remedied 
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whenever it occurs. Unfortunately, as we wilt dentorstrate in or 4, seat alone’ ts 


not sufficient to perform all transactions atomically. 


A second technique that could be: used’ to’ control the order of execution of 
transactions is one that I call explicit: focking. “Rixplicie 4ocking consists’ bf postponing the 
reception of some class of message by a process uritil some ‘otfielfriessagé has been recefved. 
Chapter 4 will discuss focking in greater detuil atyd-wilf introduce a triechanism for — 
sian into the pipes t model, 


A synchronization scheme using exp ein d does not achieve the goal of ‘Partial 
operability. Using explicit locking, a data manager could postpone the reception of a request 
“te perform some focal transaction until that dati’ ‘tanager had received other messages. 
” Explicit locking coutd cause the local transaction to be detayed “mous ns 


oe bes 


Sequencing and explicit locking both contol the order of processing operations by 
controling the order in which messages are received by processes. Another approach to the | 
control of the order of execution Of processing operations Is to control What'actlon is taken by 
@ process on receiving a message. The il iad seaport aie use this 
approach. 


One way in which. a 1 process can ‘postpone the  procesing operation, requested by a 
message that that process receives is to record ithe message i in the boaal state of the process. 
The stored message can be retrieved and acted on in a later process process step. One could 


_ call this technique squigreling. 


Using squirreling, a transaction local to one dati ihanager can be delayed indefinitely 
because the request to perform that transaction can be iquirreled away iridefinitely by that 
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data manager, pending reception of some other message... Tien peeling om ont. ace 
eer E nee ee 


Another mechanism that can be used. to-postpane. the, processing requested by a- 
message is.to have a process that receives a message that the precess should n 
send the message to another process. That other process woyideithes 
Pass it Gn agaltie Poesy. nck, 20: tee: firs FHC, ‘Thin, achetque coukd be refered to as 
buck passing. Buck passing also does not achieve the goal of partial operability, as a 
pegs stone = ec Sees Sabaneta By be psa niyo 


‘to process. 


| Both buck passing and. squirreling are what .could be. called umplicit Ipgking 
(because request messages are not explicitly postponed, but the requested. processing is 
pompones) Implicit locking is characterized by the fact that two or more process steps of the 


_ data manager receiving a request are used to perform the | en ule gene by a reeneage, 


When ie OF more proces aps of & angle daa manage are wae to carry out a 
transaction, the gosl of partial aperability.is not achieved, If two or more process steps carry 
out accesses for a transaction, then other transactions that access the items accessed in those 
steps may have-to be exchuded from occurring between the two steps. If a failure delays the 
” second step ‘of a data’ manager ‘performing ’ ‘accesses for a ‘trinsaction, then transactions local 
“to that data manager that must be excluded may be indefinitely delayed. If only one process 


step (of two or more) of a data some plicit i ln acto, then some 

formedby the first ‘step of the 
data manager. The manager must in effect be waiting for somm. mesage before it will 
perform the accesses for the. transaction. That message. could. be delayed. indefinitely, 
delaying the transaction. 


condition must be preventing those accesses from being | 
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To summarize, the sequencing mechanism is the only one of the techniques for 
controlling concurrency in the process model thai i¢hieves the goal of pattial operabitity: In 
Chapter 3, we will demonstrate a mechanism that woes sequencing to provide contro! for 
‘Many processing operations. In Chapter 4 | demonstraze that sequencing alone is insufficient 
for coordination of all possible transaction, and show that some Aechanism in which two 
process steps of some process are used to perform one transaction is needed. | 


ge a TR Ee. ; ‘es 


2.3 Summary 


This chapter Presents a semantic model for a distributed information system in which 
"the effects of failures are well specified. The model combines features of Actors, Data Flow, 
and the Object Model. The model makes a strong distinction between two forms of 
. communication: inter-process messages, and intra-process communication through shared 


state information. 


| Two different classes of failures in a distributed information system were discussed: 

site failures and communication failures. We showed two ways in which the process model 
could be extended in order to include a specification of how computation is affected by such 
failures. One extension (simple processes) was easy to implement, but allowed failures to 
have relatively severe effects. A second extension (robust sequenced processes) limits the 
' visible effects of failures, but requires more overhead in its implementation. The remainder 
| of this thesis will make use of robust sequenced processes in developing algorithms for 


_ performing transactions. 


The problem of performing transactions atomically is translated into the terminology 
of this model, and a plan for an implementation of a distributed information system based 
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on the mode is given. A condition for determining whether or not a transaction is atomic is 
expressed in terms of the ordering relationships of the model. 


Finally, techniques for controlling the order of execution of process steps were 
discussed. One of these techniques (sequencing) was shown to be consistent with our goal of 
- partial. operabifity. Other ‘techniques allow & failure to delay the completion of a local 
__ processing operation indefinitely, but as we shall demonstrate in Chapter 4 are necessary for 
coordination of some kinds of transactions. 
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from the execution history ef # For a process that receives messages at one port only, the 
_ execution history cam be represented as list of meunges received, as shown in Figure 3 
| This representation can be viewed as a log, recording each message received by p as it is 
received. The most recently received message in the execution history is at the top of the list. 


I define a broadcast message to be a set of messages and destination ports. A 
broadcast message B can be represented by a set of pais imp Uy auch that mAs 
| message, and py 1s the name of the port and proce) te which my eset. The individual 
messages that make up a broadcast are referred to aa the gomnamaamts of the broadcast. The 
order in which a group of receiving processes receive a group of broadcast messages can be 
derived from the order in which the components of thove broadcasts are received by the 
“individual processes. 1 The order of two broadcast meuages By and Bp, is defined as By < Bp 
if By contains a message my sent to a process p, and By contains a message mp also sent to p, 
and m <, Mo. This: definition: is completely analogous. tc the defiaition ef she .« ordering 


Figure SA - 
The Abbreviated Execution History uf a Process: 


ee har oe ae 


AM <p Mg <p My 
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Chapter 3 
Atomic Broadcasting = 


Se Gage eer 2 


‘Many transactions performed iy a distributed information system can n be Grampa 
into > independent component operations, each of which he performed at one ate and does not 
: depend on end other site. In the model of the previous chapter, cach component of such a 
transaction is performed by a single process step. Al of the ne that form the inputs to 
; these. process — can be constructed in advance, before on sep is performed. The 

ordering of such a transaction relative to other transactions _ controled wd ue order in 


4. 


which these messages: are received 


of Reso 


In ae chapter, I-introduce a mechanism:for stomic pres@eastiag, which distributes 
| a set of messages to a set of destination ports 3 that they are received atomically with respect 
to ether such sets. If an ese broadcast is use to distribute the input messages for a 
transaction with independent components, that.tgantigien is. performed, atostically. Atomic 
broadcasting isa simpler problem than that of coordinating arbitrary transactions. | 


8.1 Definitions 


For convenience, I assume that all messages and all ports are uniquely identified. 
Many processes receive messages at a single port only. For such processes, I will use one 
identifier such ai p to refer both to.a process and the port at which that process receives 
messages. Recall that for each process p there is an ordering <p on the messages sent to p 
that reflects the order in which those messages are received. Each message m is included in 
the order <p when it is received by p. The ordering <p for a process can be determined 


- 57 - 
on transactions. Similarly, a broadcast message M is atomic with respect to some set of 


broadcast messages if the < ordering on those messa 


Fieure 3.2 illustrates the reception of three broadcast messages. that were not atomic. 
B, includes two component messages mx for X and my ‘tor '¥. ‘Similarly, ‘Bo and By 
contain components for X and Z, and for Y and Zz respectively. ‘In this ‘example, “X receives a 
‘Comporient of By before one from Bp, V receives a component of By before one from Bp, and 
7 z receives a Ect of Bo before ae These ordering Fealnehips constitute a age 


rT be. iuaaiee! to be ordered is | 
if the sender of one message was one of the receivers of the ether. For the moment, I. will 
_ ignore this kind of ordering relationship. A later, section extends, the notion ,of, an_ atomic 
broadcast described here to include such relationships. 


A second way in which two broadcast mes: 


Figure a2 
Non-Atomic Broadoasting 
| ; By {my y X>,<my y.¥>} 


Ba » {cme x X>.<mg 7D} 
Bg = {<img y,Y>.cing 7.2} 


Ot 


Bo > By By; > Bs Bg > Bo 
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3.2 An Llustration of Atomic Broadcasting _ 


The independence of the process steps to be coordinated in’ in atomic broadcast (the 
steps that receive the messages that make up the broadcast message) makes coordination of 
Atomic broadcasts simpler than coordination of more general operations. A simple real workd 
analogy may help to illustrate this point. Consider an office, in which ail communication is 
_ through interoffice memos. Sending some important notice to all smployees about a change 
in working procedures. is an instance of an atomic broagicast.. The notice should be sent 
atomically, so that employees working on and communicating about the same project receive 
the notice at the same poiit in their work. This tari ‘be iictorplished’ relatively easily 
“through the office mail systern. At one instant, all of the notices are entered into the mail 
system and take their places in the queues of mail’ waiting to be delivered to arid read by the 
employees. After that, each employee will firid the notice ait the sand potrit (retative te other 
mail) in. his list of messages. It does not matter that some employee on vacation may not see 
the memo fort meth more, the wires nthe proper aquenc elatve to 


other mail. 


Compare this situation with that of a group project, which requires a joint discussion 
by a group of employees To complete such a projec atomi: - ty with respect tp other werk 
in progress effectively requires that each group. member set aside a certain time for the 
discussion. Scheduling the meeting is a much more difficuk probtem than placing a notice 
in each employees in basket. A second, more serious problem is that if the meeting has to be | 
suspended for some reason, the members of the group can not work on: any other project that 
may conflict with the group effort, as the effects of such work will mot be known to other | 


members of the group. 
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This -analogy is “crude, but’ gives a feeling of the differences involved. The 
‘distribution of the memo as an atomic act fs easy; Beaduse theré are nd edtistriaints on when 
“the recipients’ actually read’ the ‘memo. “It is sufficient to ‘phite ‘thé! theo in the correct 
“Sequence in each employee's mail. 


3.3 A Mechanism for Atomic Broadcasting — 


~ In this section, I present a mechanism for coordinating atomic broadcasting that uses 


robust sequenced communication between processes to distribute the Coniponent messages of a 


“ broadcast message t6 their destination ports. The solution “uses! processes that F reféfto’ as 
message forwarders fo distribute these messages. Each message Forwarder receives messages 


- ata single input port. A message forwarder has a single process step specification Which can. 
be described bya function JAM) = itm,pl}, mapping each — eds to a set of ogus 


%e gee” 


- messages a and destination’ ports f for those |  tnssages. 7 


The Beets received. or sent by a message forwarder .each contain a set of 
component. messages and destination ports. The components of each such message form. a 
‘subset of the messages that comprise some atomic broadcast. Each process step of a message 
forwarder receives some input message ad paititions thé components Of that mésiage among 


Wyott 


the output messages that it produces. For each’ such ‘step, thé ‘output 


contain eee the same set of af canine as the ees aeecs to that one 


ie 


The ca for. atomic pace organizes. a of 4 the, -processes. -in the system, 
processes, and data managers), in ablerarchy.. Fach process 
p has a unique parent f in the hierarchy. | will also describe this relationship by saying that 


| (message forwarders, transactic 


pf is a child of f. I say that p and q are relatives if either p is the parent of g or q is the 
parent of p. In the hierarchy used for this protocol, each process f that is the parent of some 


- 60 - 
other process is also a message forwarder, and there is.a.single message forwarder r which is 
the root of the hierarchy, and is an ancestor of all other processes. The transaction processes. 


and data managers. form the leaves of this hierarchy.. Any hierarchy of message forwarders 


can be used to perform atomic broadcasting. As we shall see, bawever, the.organization of 
the hierarchy determines the number of messages that must be sent to distribute each 
broadcast, and should be made with some knowledge of the. expected communication 


patterns. 


- In order to send an atomic broadcast, a process formulates a single message 
containing a set of components, each of. which specifies a message to be sent and a 
destination port. This single message is sent to any message, forwarder that is above all of 


_the destinations in the hierarchy. Recall that each step of a message forwarder partitions the 


components of the message received among the output messages produced. Each message 
forwarder sends output messages only to its children in the hierarchy. On receiving a 


Message, a message forwarder partitions the components of that message such that each 


component is sent to the child that is above the destination port of that component in the 
hierarchy. ss Sid rs re 


_ Figure 3.3 illustrates the operation of this protocol in distributing the three broadcast 
messages shown in Figure 3.2. The processes are organized in a three-level hierarchy, where 
f is the parent of processes Y and Z, and 1, the root, is the parent of f and X. Figure 3.3 a 
shows the orderings for all processes after B, and Bo have been received by 7 and By has 
been regeived by f. Figure 33 b'shows an intermediate state ih thé distrtburtion of messages 
to X,Y, and Z. Figure 33'c shows the final state when all compotients of all'three broadcaists 
tiave been received. ee . aa 
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Figure 3.3 
Coordinating Atomic Broadcasts with Message Forwarders 


By = Im, x X]im, y.V}} 3. Sa 
Bg = {mg x.X Hm 7.2)} 
Bg s may img pei The Initial Bxecution State 
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Figure 3.3b 


An Intermediate State 
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Figure 3.30 __ 
The Execution State After Delivery of By, Ba, and Bs 
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A brief argument for the correctness of the solution is given here. A more detailed 
‘and more formal proof appears man appendix to this thes a eee 
. B, and By are initially sequenced by one mesiage message forwarder, the highest message 
forwarder receiving messages: connected with both broaccasts.- Because the message ‘stream 
between any two processes is sequenced, the order of two breudcasts sequenced by a message 
_ forwarder is preserved as the messages connected with those. broadcasts travel down the 
hierarchy toward their destinations. This sequencing. iopures that no pair of broadcasts in 
the < ordering can form a cycle. (ie. there art mo message for which By < By and Bo < By). 


The proof that na larger cycles can arise is substantially more complicated. The 
proof of the message forwarder protocol given in the Suceiiia Wuéers cycles of all sizes. 
This proof uses the properties of the hierarchy to show that no cycles can be achieved 
without.a violation of sequencing between a process and its parent in the hierarchy. 


There are several desirable properties of this protocol that are not obvious. One is 
that each process executes a single process step for each broadcast. This mechanism does not 
‘use locking as defined in Chapter 2 ‘The sohution insures that al! processes, including the 
‘Message forwarders, receive the messages, ‘ent int distributing a broadcast- message atomically. : 
The transaction " synchronization mechanism described in the next chapter malies use of this 


property. 


Another point to note is that the protocol works for structures of message forwarders 
other than hierarchies. I will use the term synchronization petwork for the logical 
_ organization of processes used in the protocol. A synchronization network is simply a 
directed graph that describes the parent - child relationships among processes. The proof of 
the message forwarder protocol given in the appendix depends on the synchronization 
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network only in that it requires that there be at most ‘one path 4n the synchronization 


network between any two processes. This property is,.of course, satisfied by a hierarchy. 


A second requirement that must be imposed on the synchronitation network used in 
the distribution of a broadcast B is that the destinations of the components of B must have a 
common ancestor in the synchronization network. If this were not the,case, there would be 
no way. to distribute B using the protocol, because there's no’ process to which B can be sent | 
A initially. If we are designing a synctiregiestion network capable | of Coordinating any 
r broadcast message involving a group of Processes, then we must insure that all of those | 
processes have a single common ancestor. _ This requirement, taken. together with the 
requirement that there be at most one path between Processes _ in the network, _Mmeans that 
_ Such a synchronization network must be a hierarchy. If, however, the set of broadcast 
messages (or at least their destinations) is known, and a synchrohtiation network is being 
designed specifically to distribute those messages, then it is possible that a non-hierarchical 

network could be used. This is itfustrated by the example ni Chapter 6. 


“3.4 Other Orderinig Restrictions on Broadcast Messages 


| _The above protocol insures that each process receiving a. component of a broadcast 
receives that component in the same order relative to, those. of other breadcasts. Thus a 
broadcast is atomic as viewed by the receivers. Recall, however, that there. is another, way_in 
which the processes may perceive ordering among broadcasts, in that the sender of one 
broadcast may have been a recipient of other broadcasts. 


In general, each process step that produces a broadcast may have received some 
knowledge about other broadcasts. This potential knowledge can be described by the should 


follow relationship among messages described below. Each message m sent by a process p in 
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a process step s should follow a message m’ received by p whenever: 


a) There is a message m” received by # in process step s or in a step 
that preceded s, and m’ and m” are components of the same 


OR 


sy Peas td ety ete By p96 2p ta 
preceded s, and m” should follow m’. 


This relationship describes ordering constraints among messages that must be enforced in 
order to prevent the system from behaving anomalously if the correct interpretation of a 
message m sent by @ process p to a process q depends on ¢ having received messages 
containing information that was derived from broadcasts feceived by p before p sent m. 
For example, if L.could in one atomic broadcast send my paycheck to be deposited at . 
the bank and checks drawn on my account to pay monthly. bills, would be disturbing to me 
if when one of those checks was sent to my bank to.be cashed, it: arrived before the deposit. 
This kind of behavior does not violate the definition of an atomic broadcast given above. 
In this ‘example, there are (wo separate actions: my distribution of. the deposit and payments, 
and my creditor’s sending of the check ‘to the bank to be cashed. Each of these could be sent | 
ina separate atirnie broadcast, however they cannot be part of the saitie atomic broadcast, as 
the debtor's action is not known until the check is received. Nothing in the definition of 
‘atomic broadcasting prevents these two broadcast messages froin’ being sequenced in the 
apparently anomatous order, ‘because the causal > between the two events that 


’ 


produced these broadcast messages is not recognized. 
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Unfortunately, the protocol described above allows such anomalous sequencing to 
occur. Consider the hierarchy shown in Figure 33. A message m,sent.to both X and Y- 
must be initially sent to the message forwarder r. It is possible for X to receive its 
component, and construct and send a message to y, and haye this new message received by 


Y, before the component of m sent to Y is received at Y. 


A simple extension of the message forwarding system described above provides 
correct sequencing. Each broadcast B must initiafly be sent to a ‘message forwarder f in the 
hierarchy that is an ancestor of the sender of B as well of as-Ail of the procesées associated 


with the destination ports of B. 


Notice that if a component of some broadcast B has been received at, any port, then 
any component of B that As destined for a process p and has my yet been received ‘by p must 
’ be awaiting ‘reception at the input port to some process that is an ancestor of p. The 
= extended protocol prevents anomalous _ Sequencing by insuring ¢ that a message B enters the 
hierarchy above all of the messages that B should follow. The sequencing of meee 
between the message forwarders then insures that any message that B should follow will be 


received at its ultimate destination before B. A more detailed pu appears in the appenere 


This solution to anomalous ssnincla is very ump (hough t the proof that this 
solution works is somewhat complicated), and easily implemented. Therefore, I will only 
consider the implementation of the more complete solution. 
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3.5 Implementation 


In this section, I wifl present two simple implementations of the synchronization 
protocols described above, one using point-to-point communication, and one taking | 
advantage of communication technology that makes distribution of one mestage to several 
‘receivers relatively inexpensive. There are many optimizations that could be used to 
improve these implementations. | present them merely to. show that such 2 system could 
easily be implemented, and that I have not ignored any difficuk problems by making 
_ unreasonable assumptions abput the implementation of message forwarders. 


3.5.1 Atomic Breadcasting Using Point-to-Point Communication 


In chapter two 1 present simple mplemeitaon of robust sequenced 
communication. This implementation can be extended to implement message forwarders. 
Robust, sequenced communication insures that messages sent from a méssage forwarder to. 
some port arrive in the sequence in which tty Tecra ‘prided, ind stk te In addition 
"Proper sequencing, we emt show how the hirshy of eemage, forwarders can be 


‘maintained. 


Xi Gute ar dnc shor ois idan 
each process that sends a broadcast message must know the location, in the hierarchy, of 
each of the destinations of the components of that broadcast. isk isibohatge a neceiaazy v0 
select a message forwarder that is above all of the destinations. A second problem is that 
each process may send messages to a large number of ports. This is expensive using the 
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implementation of processes described in cans 2, because nemere aoe must be 


maintained for each such port. 


_ [solve these problems by changing the protocol for distributing the components of a 
_ broadcast slightly so that each process need only comminichie’ with its parent and children 
in the hierarchy. This is accomplished by changing the process ‘step specification of a 
message forwarder so that if all of the comporients ‘ofa message received by a message 
forwarder are bound for descendants of that message forwarder, the message is partitioned 
among the children of the message forwarder as-before. If, however, the destination of some 
component is not a descendant of the message forwarder, the message is sent, intact, to the 


parent of the message forwarder. 


To send a broadcast using this modified’ protocol, a process formulates a message 
containing a list of the component messages of the broadcast, and sends that message to its 
: parent in the hierarchy. This message rises in the hierarchy until it is above. all. of the. 
: destination ports of the components of the broadcast. (as. well as its sender). When the 
pee reaches a message forwarder that is above all of the destinations, it is distributed as 
_ before. Each process communicates directly only with its immediate neighbors. in the 
hierarchy, thus the number of message queues needed to maintain robust sequenced 


communication is small. 


_We can now consider how the necessary information ‘about the hierarchy coukd be 


~ maintained. “Each message forwarder f must know which ‘processes lie below ‘each of its 


eit see 


children in the hierarchy. This knowledgé could be built into each mm forwarder, or be 


built into the structure of process names. If the life of the hierarchy exceeds the usefulness of 
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individual processes, however, we must expect that processes will be created, deleted, or even 
_ Moved in the hierarchy, and that these changes must be reflected to the message forwarders. 


In showing how to add a process, I will assume a mechanism for generating unique 
Process.nammes, and wil not attempt to solve the problem thet a process being added to the 
_ hierarchy may already be part of it. I also assume that it is possible to determine from a 
. port name which process receives messages at that port. This knowledge allows the message 
forwarders to determine the destination process of 2 message from its destination port 


To add a process p to the hierarchy, some message forwarder f is selected to be the 
parent of p. Process p informs f of this choite by sending a “request for adoption” message. 
This message establishes the message queues and sequence nisnbers for ‘sending messages 
| from p to f. Message forwarder f can reply to p either by seating or. rejecting this request. . 


If the request is accepted, the mechani for sending messages from f to p is 
established with the sending of the reply, and p can begin to ‘send and receive atomic 


Broadcast messages. Message forwarder fsendsa message to its parent whict is propagated — 
up the hierarchy informing all processs that are now ancestors of p of the presence of #. 
Before any message can be sent to p, the sender must be informed of the existence of p. Any 
message that could inform a process of the presence of p must either have been sent by p or 
should follow (as defined in the previous section) some message sent by ». The messages 
that inform the message forwarders of the presence of p will always precede any message sent 
by p (and therefore any message that should follow a message sent by f) at the ancestors of 

‘p, because of the sequencing. Thus any message forwarder encountering a message with a 
component sent to p is guaranteed to know whether or not p is one of its descendants. | 


ae 

Special care must be taken when ‘the request for adoption is. rejected. 
Communication failures can cause either the request ‘for adoption or the reply to that request 
to be lost. We must be sure that loss of messages cannot cause p and f to become confused 
such that one thinks that the aie ae successful while the other does not. Such confusion 
: is particularly: likely if the request is retransmitted by p if f does not respond promptly. 
This problem is similar to that of initiating a connection in a communication protocol, such. 


as TCP (Cerf74] or DSP [Reed?6]. The solution that I am using is similar to that of DSP. 


_ When . message forwatder rejects a request for adoption, it may be doing so because 
it has insufficient resources to establish communication with a new child. If this is the case, 
we do not wish to burden the message forwarder with the task of remembering that it has 
rejected a request. Therefore, we must keep in mind that if a message forwarder f is sent a 
request for adoption several times (because the sender of the request _re-transmitted the 
request when f did not reply promptly with an acceptance), then f may first reject the request 
and subsequently accept it. (Once a request has been accepted, however, the message 
forwarder can’ know to accept any subsequent re-transmissions ‘of that request). This means 
that a process that has sent a request may not negotiate with another potential parent if it 
receives a rejection ¢ or + nO prompt reply. If the original request or a re-transmission of the 
original request) were later accepted, this coukl allow one process to have two parents in the 
hierarchy. Thus we require that if a process receives a rejection (or no reply at all), it must 
either keep trying (re-transmitting. its request) until it is accepted, or choose a new unique 


name and attempt to establish communication with another paren 


This ie may result in a message forwarder pee a process that no longer 
exists (because that process has chosen a different name), but this does not cause a problem. 


No messages will ever be sent to or from such an abatidoned process. An abandoned process 
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will be detected and deleted from the hierarchy through, the same mechanism that deletes 
"processes that is described in the following paragraph. 


A process can remove itself from the hierarchy by sending a message to its parent 
notifying its parent of its intention to leave the hierarchy. When a message forwarder 
receives such’ a request, or a message forwarder can reliably determine that one of its child 
| processes no longer exists then that message forwarder can reclaim the message queues for 
that process and inform the ancestors of the message forwarder of the disappearance of the 
-_ process. Once a process has left the hierarchy, it must chanse a new ame in onder to re-join 
_uniess it can be determined that no process remembers the old name. 


A process p can be moved from one location in the hierarchy to another focation in 
the hierarchy in a series of smail steps of the form shown in Figure 34. Each such step 
changes the parent ofp from fim g, where fh the past of g of the parent off Both 
ng eee 2 1 wh Sy Scat abe 7 


. iia wall Sakae As ceca ws cama 
Renee ee oes Menmmes re: 


sii scabegh tls icpsse wid arbi’ tn chomlc acidic tes 
state with the following changes: A reqiléit to Close ts put at the end 
of the output message queue for p, os request is put in the output 

menage qutun for, und fi View at toe 3 ectecreclale 
ass eoamae : 


3) pp receives the request for close from f, drape its new empty queue 
of messages for /. p now sends a request for adoption t0 ¢, 
dig alan coe Vetaied Wein 7 ecstalies A gae/ fia p and 


accepts p's request for adoption. 5 tee Ngee er ee 
hierarchy to incinde p 


The last two steps take Siac gaits order, depending .on. the relative timing of the 
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Figure 3.4 


Moving a Process 


From: 
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messages sent from f to p and g. No knowledge of,the move must be propagated beyond 
and g. ee ee ee ar eee eee re ee es 
refusing the request for adoption. 


$.5.2 Atomic ee with a Broadcast Medium 


In many communication architectures, tis no more couly to send a message to a set 
of receivers than to a single destination. A broadcast metwerk such as a ring | network 
(Farber72) or an Ethernet [Metcatfe76) has this property, as does ‘communication through 
shared memory on a single site. Our scheme for atomic broadcasting can be modified to 
take advantage of this ability to distribute component nme: of a broadcast to several 


receivers. 


In the absence of errors, a broadcast network acts like a message forwarder. Each 
site presents its messages to the network. The network receives one message at a time, and 
distributes that message to the intended receivers. emer: sent through. the network are 
totally ordered, just as messages sent through a forwarder. 


To send an atomic broadcast message to a set of receivers on, the same-network, all of 
the component messages of that broadcast are packaged into a! single message for the 
network. If the packet size of the network is too small to hokd all of this, the contents of each 
component message can be pre-distributed to its intended receiver. The sender picks a 
unique identifier for the broadcast and attaches it to each component message, sending the 
component messages singly or in groups to the intended receivers. When such a component 
| message is received, it is saved by the receiver and not placed in the stream of incoming 
messages. When all components have been distributed, the sender sends a message 
containing the unique identifier to all receivers using the broadcast capability of the 
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. communication network. The unique identifier is used by the receivers to identify the 
component message that was pre-distributed and insert that message into its stream of 

incoming messages. The broadcast networks designed thus far all have a packet size large | 


enough to accomodate such a unique identifier. 


If very large messages are sent, it would seem that.we,are not sbtaining any. benefit 
from the availability of the broadcast mechanism over the point-to-point scheme described 
above. Note, however, that if we were to use the point-to-point scheme for Sooroinang 
atomic broadcasts among the processes executing on sites connected by | a. broadcast network, 
_ then in gaat each component of a broadcast message would have t to dues transmitted over 


var 


2 destination. The Jain of this section transmit each component of a broadcast message 


exactly once,’ thus mee extra message transmissions. 


If the network and sites were completely reliable, including all components of an 
atomic broadcast in a single message would be. sufficient to..distribute the component 
messages atomically. Unfortunately site failures or simple lack of buffering can cause.a site 
to miss a message from the network. To solve this. problem, there must.be a mechanism that 
uniquely orders the broadcast messages, even if failures.occur during the transmission of a 
broadcast. Such a mechanism would allow each site to. know the. order in which incoming 
“broadcast messages should be processed by that site, even if failures cause some of the. 
transmissions to the site to be lost or to arrive ci of sequence. A mechanism must also be 


provided to allow a site that has missed a message to obtain a chpy of that message. 


1. This excludes re-transmissions necessitated by errors. . ° 
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This can be ‘accomplished by appointing one site as the coordinator of the 
broadcasting. The coordinator has the responsibility for arbitrating the broadcast messages 
on the network, and does so by assigning a sequence | number to each. To send an atomic 
" broadcast, a site assigns a unique identifier. to that broadcast and transmits the components 
_of that broadcast in one or more transmissions on the broadcast network. Each transmission 
_is identified with the unique identifier, so that the receivers can’ identify the transmissions 
that are used to distribute an atomic broadcast. | | 


-These transmissions are seen by every node ¢ on the broadcast n network, including both 
the recipients and the coordinator site. Each recipient | receives and stores its component of 
‘the broadcast from the transmissions used to distribute that component. This stored 
component is not yet included in the input menage queue of the receiving process at that 
site. The coordinator receives and stores alll of the components. When the cpordinator has 
received all of the components of an atomic broadcast, the coordinator assigns a sequence 
humber to that broadcast and transmits a mestage to‘all sites containing the sequence 
number, the faihie of the sending site, and ‘the ‘seriding site's unique identifier for the 
broadcast. This message informs afl receivers of components of that broadcast of the proper 
sequence in which ps een So parce eet ae pape eee eee 


ee ees yee 


The message from the coordinator also serves a3 an’ 
broadcast that the broadcast has béen distributed ‘rid the sender can delete it from its output 


message queues. 


It is relatively simple to see that this scheme works if no errors occur, as it is 
essentially the same as the scheme for distributing large broadcasts in the absence of errors 
described above, with the exception that the coer diaas aistriburas the single message that 


demands all receivers to include the broadcast in their put — queues, rather. than 


ar 
& 
wd 
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having the sender of the broadcast do this. To see how this scheme also works in the event 


‘that messages are lost on the broadcast network, ab escicamorcns 


One error that can occur is that one of the transmissions used to distribute. the 
components of the broadcast is not received by one or more sites. If it is the coordinator that 
misses one of these transmissions, then the coordinator will never detect the broadcast as 
being complete, aid wilt not send the sequence number meéisige. After a suitable timeout 
interval, the sender of the broadcast can detect thiat sothething is amiss’ (because it does not — 
receive the message‘from the coordinator) and can retransmit the components. Any site that 


received the components correctly the first tiie can: ideritl 


y arid’ discard the retransmission | 
because of the unique identifier assigned by the sending site. 


_ If one of the receivers fails to receive a compotrent. correctly, ‘but no-other errtrs 
occur, then eventually the coordinator will transmit the sequetite' namber for the ‘broadcast. 
The recetver' will discover that it has not stored tie component Yor the ‘broadcast identified 
in the message sent by the coordinator, and’tan request retransmission Uf that component‘by 
the coordinator. Thus the coordinator also acts as a backtip for Obtiining copies of lost 


messages. 


Another error that can occur is that the the Memage sent by the coordinator may t be 
missed by one or more sites. If the sender of fhe broadcast does not see ¢ this message, it will 
begin a needless retransmission, which again can be ‘discovered and discarded. by the 
receivers. The coordinator can retransmit its message to acknowledge the distribution of the 


broadcast to the sending site. 
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If one of the receivers misses the coordinator’s message, this.may not. be immediately 
detected. The receiver will detect that it is out of date. when it next receives a.message from 
the coordinator. That receiver can then fais retransmission of the =e that it has 
missed from the coordinator. 


The protocol described above for atomic broadcasting using a broadcast 
‘ communication network is. relatively simple, makes efficient. use,of the netwerk if no. errors 
occur, and works correctly if. messages are lost. or. duplicated by the network. There are 
‘several points about this. protocol that.must be clarified: before.ie.cap be.used.as the basis for 
a practical implementation, of atomic broadcasting. = 


The coordinator site must récord aff of the bréadcast messages, and must keep each 
- broadcast untit it knows that that broadcast hasbeen repeived by all receivers. In order to 
avoid having to save broadcasts forever, we cam have.each. site periodically senda. message 
containing the sequence mimbér ofthe most recent broadcast that. that, site has received 
_eorrectly. The coardinater can use these messages to, determine. when it.js.safp.to delete a 
“saved broadcast message, and when a site. is out of date.and, should: be sent, information 
"about one of the saved broadcasts. The message sent by the coordinator must identity which 


of the sites are receivers of the broadcast. This information can be devermined from 


aa * he & SE Magog 


‘in the message by using a 
‘bit vector with one bit for each site indicating see not that site is a receiver of the 


‘examining the components of the broadcast, and ca 


broadcast. The bit vector is used by a receiving site in order to determine whether or nct 
that site should have received a ‘comfiponent of the broadcast. This in turn tells the site 


whether or not it missed the transmission of the component by ‘thie sender. 


ee ne 
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Each site must keep track of the most recent sequence 1 nome sent by the coordinator 


_ that has been. seen and correctly processed: by: the: site.:: Ina typical application of this 
protocol, it might be-the case that cach site is a ecaitec in relatively few of the: atomic 
“broadcasts... If this-te the case, it may. be necessary to filter thecmessages sent by the sender 
, and. by. the. cagrdinator in. the receiver's. network nterfage.dn order to aveid: interrupting the 
_. receiver. unnecessarily, This could. be done by. apaintaining::a register. in. a site’s network 
_ interface, which contains.the.sequence nuspber: of. the mest recent-broadcast-that that site has 
correctly processed. When a message from the coordinator is seen by the network, interface, it : 


examines the ines aee to determine whether or not the sequent number in that message is 


are 


“one greater than that in the register in the network interface. ‘If this Is so, and if the ‘Message 


‘does not describe a broadcast in which the site ts a receiver, then the register is incremented, 


if 


: and the receiving site is not interrupted. Tf a menage from the ‘coordinator does not meet 
Wi iia: opbyAgiats 6 


; these conditions, then it is reported to the _reeiving she bidiew: either detects that the 
sequence ‘number indicates that the receiving site is out of date, or that the message pertains 
“to the receiving : site. If the message persis to , the receiving ste, then the receiving site 


‘ incorporates the broadcast described by the menage into its, input message queue (in stable 
teri 7 oa 


storage), and then updates the sequence ‘number in ts network interface! Otherwise the 
“receiver requests retransmission hes the missed message) from the coordinator. 


1. Notice that if a second message comes in before a message received by a site has been 

_ dingorporated,. the. sequence number in the network interface of the alteimay:: -be out of date. 

This causes no problem, as the site detects that it has missed the second message and 

> immediatelyobtains it:: The sequence number:cannet:be-apdgsed :before ‘the ‘message has 
been recorded in the input message queue, as a failure of the site may cause the message that 
has been received but not yet recorded in the queue to be fost. 
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3.5.3 Use of Broadcast Networks and Point-to-Point Communication Together 


The schemes for providing synchronization of atomic broadcasts usity a broadcast 
network can be used in conjunction with: the: message provétots for’ point. to ‘point 
conscnanicacion in a network with a number of different physical vonimiifiication tnedita. To 
do so. most efficiently, ail of the processes rimming "at sites hiked ‘by &-BFéideast network 
should be made children of a Soh ange Sener eens Pe eee Other 
. broadcast networks and sites are finked through message. forwitders representing ieee 
connecting networks. | 


To see how this is done, consider the physical communication apology shown in 
‘Figure 35. The physical configuration is three broadcast subnetworks, with sites F and G 
acting as gateways between Netl and Net, and between New and Nae ‘respectively. One 
possible efficient hierarchy for this network i shown in Figure 26, This figure isa skeleton 
hierarchy showing one neg? forwarder for each ste. Mees processes ata site would be 
descendants of the single message forwarder itis for that site. Consider a broadcast 
“message sent bya seabed at site gto ap heasagey at sts D, E, A, and b. Site G would use the 
broadcast network Net2 to distribute components to sites D, g ‘and ‘F This message 
forwarders at D and E would route their: components to the Proper destination _ Processes. 


The message forwarder at F would use Net3 to distribute the messages for H and . 


3.6 Evaluation 


The algorithm described here for coordination of an atomic broadcast is only one of 
many that coutd have been used for this purptise. The desirability of” this algorithm as 
— to the others naga mainly. on the: extent: te whith sed + hierarchy of mewage 
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Figure 3.5 


A Physical Communication Topology 
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Figure 8.6 
A Logical Topslogy for the Network of Figure 8.5 


forwarders reflects the logical and physical communication paths in the distributed 


__ information system. 


I have already argued that many applications for a distributed information system 
exhibit. a strongly hierarchical organization. This is a reflection of hierarchical management 
policies. If the hierarchy of message forwarders is chosen so that processes that need to — 
communicate frequently are nearly always children of the same message forwarder, the 
message forwarder scheme involves little extra message passing beyond direct communication — 


_ between. processes. 
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This is particularly true if the physical communication network is also hierarchical. 
If the physical communication network is hierarchical, (counting broadcast networks as a 
single node in that hierarchy), then the atomic broadcasting mechanism described here is as 
reliable as any other communication mechanism. ‘Each message follows the shortest path in 
the hierarchy between its source and destination. Two transmissions take place for each link 
in the hierarchy that a message traverses (one carrying the message and one carrying an 
acknowledgement) This is the minimum number of messages needed to deliver a message 


reliably, and the synchronization adds no extra messages. 


If, however, the physical communication ‘network is strongly noh-hierarchical, with 
many alternate paths between any two ‘sites, imposing a logical ‘Hierarchy may cause 
communication between some sites to be very inefficient, where a direct link between those 
"sites exists. This problem can be alleviated to soinie extent by sending the contents of all 
large messages over the shortest possible route, and sending a message header through the 
hierarchy to designate when the pre-distributed. message contents are to be included in the 
incoming message stream of the receiver, as was done for broadcast networks with small 
packet sizes. This technique reduces the communication overhead due to the hierarchy, but 
does not reduce the vulnerability of the hierarchy to failures. If much communication is 
local, however, this vulnerability may not be a problem. 


The message forwarder scheme has several advantages over other synchronization 
mechanisms for distributed systems that could bé used to coordinate atomic broadcasting. 
One advantage is its simplicity. The message forwarder protocols can all be described by 
simple statements. Each step is deterministic, and the only source of non-determinism is the 


order in which two messages sent to the same process are received. 
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. The inability of two processes to determine reliably whether or not some message sent 
between them was received does not cause a problem in the message forwarder scheme. 
Using the protocols described above, once a mesage has been sent, the sender assumes that 
that message could have been received, and does. nat take any action inconsistent with that 


assumption. Spee: eearege Bat: ewes Pe peeenaes eens ee 
through. 


Another interesting feature of this solution is that the sender of a broadcast need not 
participate in the completion of 2 broadcast. Gnce the broadcast message has been delivered 
to a message forwarder, it will eventually be delivered po all receivers, even. if the sender 
crashes. The sender of 2 broadcast cannot, however, know when, that broadcast will be 
delivered, as that depends on the availability of the mestage forwarders and receiving. ports, 
and on the order in which messages are received by.thgse ports,.The broadcasts from one _ 
sender are, however, delivered in the same order in which they were sent. 


A third distinctive feature is that the order in which a broadcast is received relative 
to other conflicting broadcasts is not determined in one decision. The decision is distributed 
among the message forwarders through which the messages of one broadcast pass, each of 
‘which performs some arbitration. In a scheme using timestamps to arbitrate between 
concurrent messages, once a timestamp has been assigned to a message its order relative to 
other messages has been fixed. Postponing this decision by distributing. it among the 
message forwarders provides greater flexibility that can be important. in. same circumstances. 


Even after some of the component messages of a Broadtast have been received by 
their destination ports, other messages from the same broadcast may still be held by the 
saitier, This flexibility is important if the communication network connecting ports 
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partitions, in that broadcasts local to one or the other of the partitions cas continue to ‘take 
"place, even if there are messages from more “global” broadcasts that have not yet been 
delivered. . The extended protocal and the implementation discussed above guarantee that 
this flexibility does not allow messages to active out of order, in, that, any. port, P receiving, a 
message. B will have received any message that the-sender.of B. could haye, been. aware, of 
before receiving B. 


The message forwarder scheme takes advantage of “ocaiity of reference” in 
communication: more effectively than many other synchronization schemes that could be 
applied to atomic broadcasting. Some schemes, such as those using timestamps, require 
| periodic communication among all of the sites. Such a scheme would be expensive for a 
distributed information system in which most operations involve only one or a few sites. 
Sending an atomic broadcast using message forwarders, in contrast, requires only the 
participation of the sender and receivers, and possibly a few additional sites holding message 


forwarders. 


One point that remains to be explored is to determine exactly what kinds of 
operations can be performed using an atomic broadcast. This question will be answered in 


the next chapter. 


8.7 Summary 


This chapter has discussed one simple synchronization problem in a distributed 
information system: that of sending a set of messages to a set of destinations “atomically”. A 
mechanism was ‘developed to provide the proper synchronization by using message 


forwarders to distribute atomic broadcasts to their receivers. The mechanism was extended 
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to prevent anomatous behaviour if correct interpretation of one message depends on prior 
reception of some message. —~ — 
implementation that was independent of the physical commurtcation’ network, using robust 
‘sequenced processes was developed. The protocols affow processes to’ be added, deleted, or 
moved within the hierarchy of message forwarders. A more efficient implementation that 
takes advantage of a broadcast communication network was also outlined. | 
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Chapter 4 
Atomic Transactions in the Process Model 


In this chapter, the problem of performing transactions atomically in a distributed 
‘information system described by the process model of Chapter 2 is considered in greater 
detail. A method is presented for describing the data flow that a transaction causes among 
the items that it seceias The difficulty of. coordinating transactions to be performed 


atomically is shown to be dependent on the interaction of their data flow descriptions. 


_A synchronization scheme consistent with the goals set forth in the first chapter is 
developed. This scheme makes use of the higrarchical-‘mechanism for atomic broadcasting 
described in Chapter 3. The mechanism is simple, efficient, and frequently avoids locking. 


4.1 Analysis of Transactions 


The techniques needed for synchronizing a set of concurrent transactions are 
‘dependent on the data flow among data items caused by performing the transactions. The 
set of input items to each transaction and the way in. which those inputs are reflected in the 
updates made by that transaction affect the way in which transactions interact. I will use an 
ss abstraction which I refer to as a transaction graph: to describe the data flow between items 


caused by performing a particular transaction. 


A transaction graph is a directed graph in which the nodes are the data items in the 
data base. These arcs show how the output items of a transaction are derived from the 


input items to that transaction. The transaction graph for a particular transaction T 
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contains directed arcs pointing at each item that is updated by T. For each such item i, 
there is an arc running from each item j such that the new value given to i by T depends on 
the value of j seen by T. 


Figure 4.1 shows the transaction graph for a simple banking transaction. This 
transaction modifies the values of three items, x, y,z The transaction could represent a 
bank's action on cashing a $50 check for a customer, where x represents the amount of cash 
disbursed by the bank, y represents the account balance, and z represents the customer’s 
“overdraft protection” loan account! - rs | | 


Figure 4.1 | 
A Simple Transaction Graph 


@®  gQ——e@ 


The Transaction T: 


Set x = x-%; 
If y < 50 then do; — 
Setz=2z+y- 50, 

_‘Sety = 0, 


else Set y = y-50, 


1. In this simple example, we assume that the overdraft protection is unlimited and ignore 
any other bookkeeping that must be done by a “real” banking system. 
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The transaction graph depicts the way in which the outputs of the transaction are 
‘ computed. The arc from y to z in the transaction graph of Ti reflects the tact that the value 
for y must be obiained before the value produced by T for z can be determined. Such arcs 
~ describe constraints on any implementation of a transaction it that the access to an item that 
is the source of some arc must be performed before the access to the item that is the 


destination of that arc! 


In the process model of a distributed information system, a transaction is carried out 
as a set of process steps. A transaction graph can be iised to construct a similar abstraction, 
‘which I refer to as an activity graph, describing the data flow among the process steps that : 
implement a transaction. Two points cause an activity dea for a transaction to differ from 


its transaction graph: 


~ 1) Several of the items accessed by a transaction may be held by a 
single data manager, allowing. all of the accesses to those items to be 
- performed in a single process step. : 


2) Some data items may be replicated, with copies. held by several 
Sites. This means that one access in the transiction graph may be 
performed by several process steps in the activity graph. 

The nodes of an. activity graph are the processes that participate in performing the . 
transaction. For each arc from an itern x to an item y in ‘the transaction graph for T, the 
activity graph contains one arc pointing to each manager that hokis a copy of y from some 
manager holding a copy of x. Arcs connecting a process to itself are not shown. If an item x 
. Which is the source of some arc in the activity graph of T is. replicated, then we have a 


choice of which copy of x to use in computing the output of T dependent on x. This choice 


1. Note that if a transaction graph contains a cycle, this. means mat some item in the aa 
must be accessed at least twice in any implementation. . 
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is reflected by the arc in the activity graph of T sopeectiog some process holding a copy of x 
to the process that holds an item whose new value depends. on x. Af transactions are run 
atomically, then all copies of a replicated item seen by a transaction have the same value, and 
thus the choice of which one to use will not effect the output values produced by the 


transaction. 


Figure 4.2 shows the activity graph for an implementation of the transaction depicted 
in Figure 4.1, in which each of the items is replicated at two,of the three data managers. 
_ The graph indicates several decisions made about. the implementation of T. M, holds copies 
_of items. x. and y. The new values produced for these items depend only on their previous 

values, so a decision has been made so that M, isto compute the-new values for its copies of 
these items from their previous ae at M;. Similarly, M> is to use the old values of the 
copies of y and z that it holds to compute their new values. M 3, however, holds a copy of z, 
but no copy of y from which to compute the new value ofr. A. decision has been made that 
M ; is to obtain this information from the copy of y heid:by Af ». | 


Notice that in this example all three manages participate. im the cannpiation of the 
outputs of the transaction. This results in some duplication of ‘effort, as, for example, both 
_ My, and M3 compute new values for x. We could have centralized the computation of the 
outputs of the transaction in one of the three managers and distributed the results to the 
other managers, which would have lead to a radically different activity. graph. 


The model of a transaction used in this thesis, in which’ ‘various parts of a 
transaction are performed in parallel, is different from the model used by many other 
researchers in which the accesses required to‘ carty out a transaction take place in some well 
defined sequence. Allowing for paraltel execution of various parts of a transaction not only 
allows the transaction to be completed faster, but abo. sienpliies. the task of. synchronization 
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because the synchronization mechanism can choose the order in which two parts of a 


transaction that are logically independent (such as ‘those performed by M, and M M2 in this 


. example) are pestormed: 


The arcs in an activity graph represent constraints on the oe in which the pose 
Steps used to perform a transaction can occur. Some step of a proces that is the source of 


one of these arcs must be completed before some step of the process that is the destination of 


that are. Recall that performing a transaction atomically. with: respect to.other transactions 
also constrains the-order in which process, steps occur. The difficulty. of coordinating a 


group of ‘transactions to..be performed atomically. depends on the interaction among their 
activity graphs. | 


_ For a group of’ transactions, we can construct a joint activity graph, which is a 
merger of the activity graphs of the individual transactions. The joint activity graph 
contains an arc between two pracesses whenever the activity. graph of some transaction in the 


Figure 4 4.2 
An. Perr Graph Fer. ae inaieevos of T 


Assignment of Items to Managers 


My: %y Mo:y.2 M3: xy 
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group contains such an arc. Each arc is cabo with the names of the transactions that 
contribute that arc. Figure 43 shows an example of such a ‘graph for three simple 


“vansactions. 


Each of the three transactions is rapes for one of the three arcs in their joint 
activity graph. This is because each transaction transfers information for an item held by 


one. manager to an item held by some other manager. 


Activity graphs and joint activity graphs can be viewed! as finer grained versions of 
the “L-U graphs used to describe transactiotis in ‘SDDuBernatelnTh | The anatysis of 
transactions th SDD-1 does ‘not examine thé derivation of outputs from inputs, but instead 
assumes that each output of a transaction may depend on any of the inputs. In fact, each 
output may. depend on only a small subset of the values read, a fact that is represented in 


activity graphs. 


Activity graphs provide a simple way of describing the way in which input values 
seen by a transaction affect the output values produced. The arcs in'an activity graph also 
describe ordering relationships among the process steps that carry out a transaction in that | 
_ the process step: at the source of some are smust be ednnplehad:refere’thé. process step that is 


the destination of that arc. 


The next section of this chapter examines the impact of the: pemens of accesses of a 
group of SEA NEBIOG?, as described by their joint activity graph, on the sdaemiaetani | 
sechniques that must be used to coordinate those transactions. 
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Figure 4.3 


A Joint Activity Graph 


Ty, 


Transactions: 
TSetB=B+A 
To: SetA=A+B 
Tg: SetC=C+A 
Assignment of Items to Managers: 
My A 


My: B 
M3:C 


- 94- 


4.2 A Simple Approach to Transaction Synchronization 


. In the previous chapter, I presented a single “mechanism to distribute a set of 
messages to a set of receivers as an atomic broadcast. This mechanism could be used to 
distribute a set of input messages to the process steps of a transaction. In this section, I 
explore the applicability of the atomic broadcast mechanism to the problem of performing 
transactions atomically. I show that that mechanism can be used only when the joint activity 
graph of the group of transactions to be performed does not contain a cycle. 


The simple synchronization scheme developed for atomic broadcasting cannot be 
used directly to coordinate @ transaction that has an ‘activity graph containing an arc. 
connecting two processes. This is because there is no way to describe such a transaction ina 
| set of independent messages to be delivered ta the data managers as an atomic broadcast. 
"The process step at the source of an arc must be completed before a message describing the 
access to be performed by the process step at the destination of that arc can be formulated. 


One might expect that the atomic broadcast protocol could be modified somehow in 
order in to synchronize a group of transactions using sequencing of messages between 
processes to control the order of process steps. If the joint activity graph of the transactions 
does not contain a cycle of arcs, then this can be done, as will be shown in the following 
‘section. If, however, the joint activity graph of the set of transactions to be performed 
contains a cycle, any protocol for coordinating the transattions must use some form of 
locking, as will be shown subsequently. | | 


48S Hyg ae oc Re Dee te 
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4.2.1 Synchronization of Transaction Groups Without igs 


_ First I will show how to coordinate a group of transactions , whose joint activity graph 
contains no cycles. The approach I will use is to modify the message: forwarder scheme of 
the last chapter to allow a process to act both as a message forwarder and a data manager. 
Such a process receives a message and produces a group of ‘Messages for its children in the 
hierarchy in each process step. The messages segue need not be a simple partitioning of 
the message received, but can depend on the local state of the-fecet vitig’ process. | 


One can perform the transaction depicted-in Figure"4.2, for example, by making 
process M> the parent of both M, and M,; in-the hierarchy. ‘The transaction could then be 
performed by. sending a message describing the transaction to” ‘M> using the atomic 
broadcast protocol described in Chapter 3. This riebsage’ propagates through the hierarchy 
until it ranches Me When this message ‘is received M5, ‘that data manager performs ‘the 
specified updates to its copies of y and z. In the same process step, M > forwards the portion 
_ Of the request relevant to M), and sends a message to M3 describing the accesses to be . 
festoumen on x and z for M3. Mp includes the current value el y in the message sent to 
M 3 . 


Including some of the data managers as message forwarders: in the hierarchy allows - 
some of the Process steps of a transaction to be pefformed before the input messages sent to 
other Steps are constructed, while retaining the hierarchical Mructure of message sending. 
Recall that the message forwarder protocol of Chapter 3 Ansures that all of the processes, 
message forwarders and data managers alike, see a broadcast as atomic. The Tequest to 
perform a transaction in this scheme is treated like an atone broadcast, and thus ts seen as 


atomic by the data managers. 
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A group of transactions. can be performed atomically with the modified message 
forwarder scheme whenever a hierarchy of message forwarders and data managers can be 
constructed so that the arcs in the joint activity graph always run from a process to one of its 
descendants. This can be done whenever the joint activity graph contains no cycles. In a 
later section, I show how assignment of data items to data managers can be chosen so as to 
eliminate cycles from the joint activity graph of any expected group of transactions. 


4.2.2 Synchronization of Transactions with Cycles in the joint Activity Graph 


If there is a cycle in the joint activity graph of a group of transactions, then there is 
no way to construct a hierarchy so that a process that is the.source of some arc is always an 
ancestor of the destination process of that arc. Thus the message forwarder protocol cannot 
be used. The following. paragraphs give an argument so support the claim that any protocol 
that correctly coordinates a group-of transaction whove jaimt activity graph contains a cycle 
must use locking. | 


Consider first a group of two transactions that form a cycle, such as T; and Ty in 
Figure 4.3. The execution of a transaction consists of a set of proves steps. The arcs in the 
joint activity graph indicate that T, must be performed by a set of process steps in which a 
process step of M, precedes a process step of M>. Similarly, in performing Ty, a process step 
of M 2 must precede a process step of M >. 


To perform the two transactions atomically, either both steps performing T, must 
precede both steps for Ty, or vice versa. To perform the transactions without locking, recall 
from chapter two that at most one process step of each data manager can be used for each 

transaction, and that the sequenceing of messages between processes is the only restriction 
| that can prevent a message that has been sent from being received promptly. | We must 
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therefore prevent, somehow, the situation that the process step of T, at M, and the process 
step of To and M > are both completed before either transaction is completed. This can be 
shown td be impossible by demonstrating that this undesirable situation can be forced to 
occur in an execution of any protocol for the synchronization of T; and Te that does not use 


some form of locking. 


Consider the state of the system during the execution of Ty in which M, is 
performing its process step of Tj. If Tg. were begun. at this point, the synchronization 
protocol must prevent the execution of the process step of M> related to Ty from preceding 
_ that which accomplishes the completion of T;. Without using locking to control the order in 
which messages are received, the only way to control the order in which M 2 receives the 
messages pertaining to the two transactions so that the undesirable order is avoided is to 
have both messages sent by M), and use sequencing of messages between M My and M> to 


force the pees to be received in the correct order. 


i to force the execution of the process step of M9 that compines T, to precede 
that that begins To, both of these process steps must be triggered by menage sent from M ;. 
This means that the execution of Ty must include two process steps of M p> one that precedes 
_ the step of M and one that follows that step. Using two steps 0 of one process to perform 

one transaction is a form of locking, therefore it is impossible to coordinate the cycle of two 


transactions without locking. 


This argument can be extended to cycles of any size by demonstrating that unless 
locking of some form is used, then it must be possible to reach a state in the execution of the 
system in which each transaction has completed a process step in one of the processes in the 
| cycle, making it impossible to complete the execution of the transactions atomicafly. We are 


left with the conclusion that some other mechanism must be needed in order to synchronize a 
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cycle of transactions. The locking mechanism used in this thesis is explicit locking. This 
locking mechanism consists of delaying the reception of ‘ mesapge until some other message 
from some other process is received.' Locking is to be avoided wherever possible, because a 
failure of the sender of the expected message, OF of the communication network, may delay 
processing of messages from other sources. This violates our goal of partial operability, as 
now a group of functioning sites cannot necessarily carry out a transaction purely local to 
those sites, because one of the processes involved int the transaction mmy be licked, waiting 
for a message from some other site. Tete at sili grt ett 
Chapter 5. : 


The particular mechanism that I will use for locking in the process model is to place 
@ pre-requisite on the process step specification of a process step. A pre-requisite is = 
predicate that may include variables in the local state of the process. A process step is not 
equisite on all 


performed unless the pre-requisite for that step is satisfied. _ By placin ap bg 
process steps that receives messages from one of the input ports of a noe one can inhibit 


the reception of messages at that port until some condition is met. ~ 


With this locking Ser ieee we can now sacl ue eeecs synchronization 
_ mechanism in the previous section to coordinate arbitrary groups of transactions. 


1. Note that in sequencing, it is Scaciiie shox ihe peoniilng of a message is i aoa ia 
only until a message sent from the same sending proces is received. 
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43 Classes of Transactions 


On the basis of the activity graph of a transaction, we can group transactions into 


sR as 


terms of the mechanisms needed. 
43.1 Transactions with Independent Components 


‘The simplest class is that of transactions whose activity graphs contain no arcs. I 
refer to ‘these transactions as transactions with independent components. A. transaction 
with an activity graph with no arcs can be performed atomically ‘using only sequencing by 
using the hierarchical protocol described in the previous section. Such a transaction places 
no constraints on the organization af the hierarchy, as any hierarchy can be used. The 
hierarchy can be chosen to optimize locality of reference, without goncern for introducing the 
need for locking in these transactions. | 


An example of stich a transaction would be a transaction which adds 5% interest to 

all of the savings accounts in’a bank. The new value of each account ‘depends only on its 

“previous value. No matter how the accounts are distributed among’ data manager processes, 

each manager can compute the new bakinces of the accounts that it holds solely from their 
previous balances! na . 

It is instructive to see just how large this class of transactions is. All “query 


transactions", which do not perform any updates to the data base, fall into this class. A 


- query transaction can always be performed by. sending out a set of roe to the data 


1. In this Simple example, I have. deliberately. ignored. other. processing that such: a 
transaction may be required to perform in an actual banking system, such as accumulating: a 
total of the accounts or of the interest paid. 
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managers as an atomic broadcast in order to obtain a consistent “snapshot” of the data base! 
Such requests can be sent as an atomic broadcast, using the mechanism of Chapter 3, in 
order to obtain a snapshot that reflects either ail or-none of the effects of any other 


transaction. The sender of the requests can then gather the mee? and use them to satisfy 
the query. 2 


A second: class of transactions that always have independent components are 
transactions that only make updates to the database. If the new. values that a transaction 
gives to items are completely independent of the previous state of the data base, such a 


transaction has independent components. 


| A third class of transactions that always have independent components are 
transactions in a fully redundant data base, such a3 that considered in (Rothnie7/;Thomas76]. 
Many. of the protocols that have been developed for synchronization of transactions ina 
distributed data base work only for the fully redundant case. This point suggests that 
synchronization of transactions in a fully redundant data base may be somehow easier than 
synchronization ina data base in which each site holds, only. a partial subset of all of the 
_data items. In fact, the fully redundant case is easier, because all of the transactions in a 
"fully redundant data base have independent components, allowing synchronization to be 
accomplished without locking. 


I. If the data needed to satisfy a query cannot be accurately predicted in Buinte this may 
be a very large set of requests. An example of such a query would be “tell me the value of 
the record that this record points at.” 

2. Alternatively, the requests can ask the managers to make copies of the data items 
invotved in the query, and the copies catt be processed ‘to satisfy the query in any efficient 
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Actually, a fully redundant data base violates the assumption made about locality of 
reference, as all transactions that update the data base involve all of the sites. All such 
_ transactions must be sequenced by the root node of any hierarchy used for the hierarchical . 
synchronization scheme! A much more interesting case is that of a data base that is not 
ii ase redundant, but still has the property that all of the input items toa transaction 


| -systern have » independant components, and may also Setncnanaet siesta 
4.3.2 Transactions With Predictable Data Flow 


A second class of transactions based on activity graphs is those with activity graphs 
with well defined arcs. I call this the class of transactions with predictable data flow. Some 
of the process ‘steps that perform such a transaction must’ be completed before the input 
messages to other steps can be produced. A transaction in this class cannot be performed 
atomically using the atomic broadcast scheme in every hierarchy, but instead requires that 
each process that is the source of some arc in its sigs graph be an ancestor of the process 


that is at the destination of that arc. 


An example of such a transaction would be the simple check cashing transaction 
depicted in Figures 4.1 and 4.2. Any implementation of this tranéaction requires that an 


access to the item y precede the access that updates the value of z. 


I. Note, however, that query transactions can always be implemented as being local to one 
site and run efficiently without locking. 
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4.3.3 Unpredictable Transactions 


A third class of transactions, partially distinguished from the second, is those for 
which it is impossible to predict which items will be accessed and how until some of the 
accesses are performed. If we were to construct an activity graph for a transaction whose 
access pattern is completely unpredictable, that graph must include an arc between each pair 
_ of managers. Such a transaction would cause a great many cycles in when inchided in.a 
joint activity graph, even though the probobiity that each arc is used in any particular 
invocation of the transaction would be small. This sugge 


ts that such transactions need 
special consideration so that they do not add to the cost of performing more predictable 


‘transactions. 


| An example of such a transaction would be a transaction following a linked list of 

records, performing some Processing on each entry in the fist. For such a transaction, it is 
impossible to predict which records will be accessed before the transaction is run. The 
transaction could potentially access any record in the file containing the linked list, and 
might sraiuted information from any of those records to any other record. 


| It should first be noted that unpredictability comes in degrees. Frequently, one can 
limit the set of items that a transaction could access, far example to the records in a 
particular file. Even relatively crude bounds can reduce the number of arcs in a 
transaction’s activity graph to the point where it could reasonably be treated as predictable. | 
The assignment of data items to managers can greatly affect the impact of unpredictable 
transactions. If all of the items that could be targets for accesses of such a transaction are 
under the control of a single manager, the unpredictability disappears. Thus if 
“unpredictable” transactions are frequent, the choice of the assignment of data items to 


managers should be made with this in mind. 
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| The three classes of transactions discussed above are'a categorization of transactions 
according to the difficulty of performing them atemieally. I'am assuming; ahd this 
assumption appears to be consistent with current aa that the most arequent transactions 
“will be those of the first two classes. In fact, in many current applications of Gistributed 
information systems queries are much more frequen than updates making. me transactions 
with independent components the most frequent. With this ievieith pe in mind, I have 
| Serle ned: a mechanism to provide correct synchronization for all three classes of — 
that is substantially more efficient and robust for transactions in the first two classes. This 


mechanism is the subject of the next section. 


4.4 A Hierarchical omens for: Transaction git dcenancecae 


vgs ete 2 GW 8S 


In this section, 1 “present a mechanism.. fer. ‘anchyoniation of Giascions ina 


distributed information system that makes extensive use of the ideas deve hoped d above and in 


Chapter 3. ane mechanisr is described in terms of on on the acc td of message 
passing that:c can occur during the execution of x trenmation, in: the ent section, I consider 


the implementation questions in greater detail. 


“Then mechanism that I will use for synchronizing transactions is an extension of the 
message forwarder mechanism described in Chapter 3. The processes are ¢ organized in a 
hierarchy ‘including both data managers, which hokd items, and message forwarders which 
merely relay messages. Some of the data managers may act 4 as message forwarders as ‘well 
Each pc in this hierarchy now has two types of ports, a front door port, and some e back 
door ports. The front door ports are “used for receiving “requests ‘pertaining to new 
transactions, while the back door ports provide a mechanism that allows & process to receive 


additional messages pertaining to the current ‘transaction without enabling reception of 
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requests from new transactions. This mechanism, together with the use of pre-requisites on 
process steps, will be used for locking. 


A transaction can be initiated by any process oY formulating a message describing 
the accesses to be performed. This message invokes a set of procest steps that together 
perform the intended transaction. Some of these process sepa are invoked by messages 
received at the front door of some process, while others are invoked by back door message - 
reception. Messages sent tothe front door of some process must follow a sieitar protocol to 
that used in atomic broadcasting: 


Messages sent to the front door may only be sent to the relatives (in 
the hierarchy) of the sender. 


A process receiving a message from one of its + children aroags the 
front door may either send the message intact to the front door of its 
parent; without modifying ‘its focal stite, or it tay perform any 


processing desired on the message and generate. messages. for. . the 
front doors of its children. 


A process receiving a message from its parent through ‘the front 
door may perform the desired: processing: and' send fiestnges to the 
front door ports of its children. 


Messages sent to the front door follow the direct route in the hierarchy between the 
same argument that was used to prove that the ae of message forwarders correctly 
synchronizes atomic broadcasts aan be used to show that the transactions are atomic as 
ordered by front door message receptions. Not all transactions can be performed entirely 
“with front door message receptions, favever: A back door message is required whenever the 
process step to be performed by one process depends on data held by same. other process that 
is not one of its ancestors. Ts orders preven the mapa inrehed ny Dyck. door more from 
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introducing ordering relationships that would ‘make. transactions ‘non-atomic, several 
_ festrictions me be apenas to back me reeennee:: 


Aaiy prota lavolved ais Gamuscio may send a message to the 
back door port of any other process involved in that transaction. 


The reception of a message at the back daor.of a process in, 
conjunction with some transaction must be preceded by the 
reseptine..ef aame: message: concersing that tranenctien at the front 


door of nat ahaa 
No steps receiving messages at the front door of &@ process can occur 
betweett the step that recetves # inelinge at the IK ‘about ‘the 


transaction and the steps that receive messages at the 
about the same transaction. " . 


| These ‘eniiions taken together. insure. that all of the.staps.of a process related to a 
particular transaction are consecutive and that. the-fizst:step. of-ench, process related, to a 
transaction is invoked. through the. frome deor. Thus the. ardering, oh transactions "as: 
observed through all message receptions is the same as that observed only through the 
reception of messages at the front door ports, and this the transactions are performed 
atomically 


‘The restrictions on back door messages require advance planning before a back door 
“so that it will not receive any messages from other transactions Before getting the back door 
message. Thus the message sent to the front door of a process that will subsequently be sent 
“a back door message must contain’a lock reituest, which cautes that process to stop receiving 
‘messages at its front door until the expected back deor message iy received. ‘The next section 
describes how the messages are constructed and routed to achieve this effect. - 
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This section discusses several details retated to the Implementation of the hierarchical 
locking scheme. Fit, 1 show how the messages nesded Ja. the. implementation of a 
: transaction are constructed from the description of the transaction. This is the responsibility 
of the transaction process, though the individuat data managers ‘mast also send. messages as 
outputs of the process stepe-thet they perform: Another iene dimeuteod te the ‘coordination of 
messages sent to the bach door ports to conform to the rules described in the previous 
_ section. An efficient implementation is described in which tock requests, can be sent to a. 
large number of processes without actually delivering the requests in. most cases. This 
implementation makes it practical to run unpredictable transactions using this mechanism. 
Finally, T discuss the: prettent of choosing: the hieritthy of protests: “This hiefarchy should 
"locality of refefence” it: the transactions to be ren, and to minémize tocking. ! 


4.5.1 Constructing the Messages 


We must now show how a transaction process can perform its function of translating 

from a high level description of a transaction to be performed into a message that will 
eventually cause the desired transaction to be performed in accordance with the 

| synchronization rules described above. I will first consider the class of transactions. with 
predictable data flow, for which it is possible for the transaction proc ) 
accesses to be performed (or at least the manager processes that perform those accesses) in 
advance. Later, I wilt show how the scheme can be extended to transactions with | 


ss to know the set of 


unpredictable flow. 
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For a transaction with predictable data flow, the transaction process can know in 
advance what accesses are to be performed and thus could formulate a message to be 
distributed to the managers Seeing those accesses. Two steps must be carried out in 
formulating the set of messages. First, the accesses wobe performed must be derived from the 
high level description of the transaction, and then the set of messages to be distributed to the 
managers must be constructed. The. first of these tasks is performed by the transaction 


_ process. 


_ The second task requires knowledge of the hierarchy as. well as knowledge of the 
transaction. We could require each transaction process to have this knowledge, allowing it to 
formulate the set of messages described below, however it seems more natural to delegate this 
task to a process in the hierarchy that is above an of the data manager: that are to 


participate in the transaction. 


The transaction process formulates a description of the transaction that describes the . 
accesses to be performed. This description may be similar to a , transaction graph for the 
“transaction. The transaction process then sends a message containing this description to its 


parent in the hierarchy. 


- Each Process receiving such a description of a transaction to be performed examines 
ne description to determine whether or not all of the data items accessed by the transaction 
are held by data managers that are below the receiving process in the hierarchy. If not,. the 
description is forwarded intact to.the parent of the receiver. If ail of the data managers that 
are to participate in the transaction are below the: receiving process in the hierarchy, the 
receiving process has the knowledge to generate the manages ‘necessary to perform the 
' transaction. | The receiving process formulates a description of the transaction in a set of 
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ere ee ere eee 


Each manager is given a description of the accesses that it is to 
Lesheated waa ee ee 


Each manager thet most produce back door mesages-is given a — 
Each manager that must produce input for its descendants in the 
hierarchy fecaue of un arc in the transection. graph) is given. 2 
description of the input to be produced. 


The proces consrucing this decpton then treats i ike a menage that it ha 
received through ts from door conning component requ tobe dsribute Each such 


"meaaage is proceed as follows: 


A process M receiving a mestage through its front door post examines that message 
to see if it contains a component for M. If not, the message is partitioned according to the 
_message forwarder algorithm of Chapter 3 and distributed to the chikren of M. licacs 
message contains a component for M, then Mf takes action on the message 


The action taken depends on whether or net:the compansnt of that.message destined 
for M contains. jock request. If it does not, then At: peefuems schatever aconss is specified (it 
_- 43 guaranteed to have sufticient information to Go se), ‘pedsibly. rodifies the other 
Components of the memmge te iackude data valees tobe passed: te cits Gescendant, and 
ice i te comps st the ag te onli searing, eee 
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broadcasting protocol. Any back door messages to be sent by: Mf are also sent by the same 


process step. 


If the message contains a lock request for M, , then: dtr-cannot. perform all of its 
accesses unui it receives additional information: Seme-ef the accesses: to be performed: by M 
depend on receiving additional information frem tome other process. | distributes. the 


components of the: message to: its chien {possibly seodifying some of the components to 
include values of date items held by M), end: ‘sends any beck door messages. that. are 


_ requested. M then. stops receiving messages at its front door ‘until necessary back door 
_ ‘Messages are received. When M receives all of the-back. door messages associated with the 


transaction that sent the lock request, poner eet eae re-enable 
message reception bee the frent door, — 


‘Some. care must be taken with back door messages: to: aveid confusion. The back. 


door messages of several concurrent: transactions. for-sqme process may become. intermingied, 
causing a back door message to arrive at a process before: the corresponding lock: request 
| The simplest solution to this problem is tw use a> separate back. door pert: for each 
" trarisaction. The transaction process. initiating a transaction: choses an identifier for each 


transaction. This identifier can be combined with:a .process:name to-form’a ‘unique back - 


door port for each transaction and nie process involved in tae transaction. A process that 
has received a lock request can then enable measage reception only through the back door 
for abe particular transaction being performed. | 


| 4.5.2 Coordination of Unpredictable Transaction 


Two problems must be overcome in applying this mechanism to transactions with 
unpredictable flow. Each process that could receive.a back door message in performing a 
transaction must be sent’ 2 tock request Because the set of data managers involved in a 
transaction carmot be predicted until some of the transaction has. been performed, all 
accesses that cannot be predicted: in: advance to be pecformed by sending messages to the 
back deor of the appropriate manager when the access te: be performed is known, The 
component request sent toa manager may caver that manager to-send anc Peceive back door 
messages dependent on the items that the manager holds and the information it receives in 
- the back door messages. Any transaction can be performed in this way. 


The second problem: comes in determining when a transaction: has been completed, so 
that the data managers sent lock requests can: release those locks. Because the set of accesses 
to be performed: is not knows in advance, a process that has received a-tock request does not 
"know when it has received all of the messages connected with the transaction that it will ever 
“receive and thus when to release its lock. Each manager must-remain locked until it has 

received all a ofthe mensages that —_ to the:transaction that sent the lock. request. 


A simple solution to the problem of determining when an unpredictable transaction 
has been completed is to have some process monitor the progress of that transaction. When 
the transaction has been completed, the monitoring process can send out back door messages 
* to all of the recipients of locks to release the locks. This strategy may sound very inefficient, 
however the next section discusses the ‘problem of distributing the lock requests, and 
describes an efficient implementation of locking for unpredictable transactions based on the 
approach of this section. 
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The progress of an unpredictable transaetion is ménitored by having each process 
that finishes some portion of the transaction report to'a monitor process: Any process can act 
~. as the monitor for a transaction, however cofnmmunication wilt probably be minimized if the 
. highest process in the hierarchy: ‘involved in. performing the’ transaction | performs: the 


monitoring function. - 


Each message (front door and: back-door) sent in perfertming a transaction carries a 


completion weight. A process initiating a transaction acbitraiity assigns compietion weights 
- to the messages that it sends so that. these weigitts sum’ ‘to one. Each process step 


redistributes the completion weight of the mestage: that receives ‘among. the messages 
_ produced by that step. No message is ever a a completion ‘weight of zero, and every 
message sent by each process is given some completion weight! If a step produces t no output 
_ messages for other processes involved in: the: ‘transaction, t-tnustead produces an output 
message for the monitor containing the entire completion weigh> received at that step. Thus 
completion weights are gradually returned to:the tansaction: monitor. process as the:vartous 
"process steps. of the transaction are completed. “Phe transaction is done: when the completion 
weights in the. messages sent to the monitor:sum to one? 


I. An optimization of this scheme would be to recognize the special case of a a message 
containing only fock. requests. In performing an’ Unpredictable transaction, many of the lock — 
requests that are sent may be completely unnecessary, and. need not be delivered before. the 
transaction is completed. We can speed up the recognition of the completion: of the 
transaction by assigning, any message. containing. enly. Ipek. requests a completion weight of 
zero. If the locks in that message are necessary, some other message with a non-zero 
completion weight -willbe forced to. wait. until the:necessary:fecks;are:veceived. ‘The mext 
section describes a scheme in which the number of messages that must be sent to perform an 
’ unpredictable transaction can be substantially reduced by postponing or eliminating delivery 
of unnecessary lock requests. 

2. The arithmetic on completion weights must be done carefully so as to avoid loosing or 
. gaining completion weight due to round off error. One could maintain completion weights 

as } rational numbers rather than as floating point solution so as to aro round off error. 
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through the hierarchy, being forwarded only when “pushed” by subsequent messages or the 
completion of the transaction setting the lock. - 


This can be accomplished by. slightly modifying the implementation of the processes 

in the hierarchy. Recall that each such process maintains a queue of messages to be 
- delivered to each of its relatives in the hierarchy. The implementation described in Chapter 
2 attempted to forward message from each of these queues whenever they were non-empty. 
One could instead construct the implementation so that messages are forwarded from a 
queue only when the queue contains a message which ts not purely a tock request. 


Consider the hierarchy and transaction Gepiceed in Figure 4.4. The transaction uses 

the values of data stored at M to update data at My. M,'s only participation is to take the 

value produced by M, and use it in.an update. This transaction: would be implemented by 

sending a message containing components for both M; and Ms. When this request reaches 

Mg these components are separated. The component for M, travels quickly down the 

| hierarchy to its destination. The component for Ms, However, contains only the lock request, 

"and will not be sent from Mg to Mg until pushed by additional requests. Thus it is likely 

~ that while M, is computing, the value to be sent “th Ms, the lock request will be held up 

awaiting delivery to. Mg. This allows M5 to continue to participate in ‘transactions local to 
the right hand half of the hierarchy while T is being performed at M;. 


This example raises another problem that must be solved: that of insuring that the 
lock request for a transaction does arrive eventually, and arrives before the back door 
| messages of the transaction. This problem is partially solved by using a unique port for 
receiving back door messages related to each transaction, as once the back door message 


‘arrives, it will wait at its unique port until the corresponding lock request arrives. It would 
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be desirable if the lock request could be delivered promptly once the transaction has 
produced a message for the back door port. This can be achieved in one of two ways. 


The implementation of a secu could oie when a message is waiting at the back 
door port and send a request up the hierarchy to forward:any fock requests. This strategy 
would be effective, but may requires additional message .sending: : If the communication _ 
network topology closely corresponds to the synchronization ‘hierarchy, = second strategy may 


. be more effective. 


If the communication network topology closely parallels the topology of the hierarchy, 

then any message, including the back door messages, must essentially flow along the arcs in 
| the hierarchy to reach its destination. We can take advantageiof this fact to provide for 
prompt forwarding of lock requests when. appropriate. : Each process that-is not-a leaf node 
in the hierarchy now has a third type of port, a pass through port. Each such process is 
_ always ready to receive messages at its pass through - port, and..pass them on: to-one: of its 
relatives in the hierarchy. The pass through ports. ptovidera:methanism to send back door. 
messages from one process to another in the hierarchy through intervening processes. Each 
such message is identified with its ultimate destination, and. sent to: the pass through port of 
‘the parent: of the see: When a process: rece Sees Wenge es its pass through port it 
passes it on either to the pass through port of one of its. relatives. or, if the ultimate 


destination is a relative of the receiving process, to the ultimate destination port. 


The pass through ports provide a Mechanism to allow each procets in the path of a — 
back door message to notice its progress. A protest cah match w tuck tequest with back door 
messages that it is also forwarding by the unique port ID of the destieintior’ back door port. 
When a process has a back door message to be forwatded to'one of its relatives, it checks its 
queue of front door messages to be forwarded te the same relative for a corresponding tock 


- HG - 
request. If such ». request is Sound, civen tee back doar message can be. combined with that 
promptly forwarded, as it ts new not solely a lock request. 


| One point of caution should be noted. If the lock request is contained. in a message 
that has bean forwarded but not but not yet acknowledged, it cannot be combined, with, the 
back door message and betle must be forwarded independently. This i¢: becaues:the process 
to which the lock request had been sent may have already received it, s0 that it is toe late to 


In the example of Figure 4.4, when M, has. finished computing the value that it 
sends to Afs, it sends it ae back deer message. This message propagates up the hierarchy 
through the pass through pers of My and M4. When My sttempts to-forward. this message 
to Mg, it notices the cossespending lock request. It combines the back.daor message with the 
lock request, and sends. the combined: message'te the front deer of M,..\When My receives, 
the combived message, it performs the specified update,-and realiacs that its role.in the 

transaction: is complans, seting and: releasing a-lock in the: sane process. sep. If; however, 
Ms were requised 2 revebwe additional mesiages in carrying oUt the, trauaction, it woud 
remain locked until those messages were received. 


Pass through ports also provide a mechanism to optimize the execution of 
' unpredictable transactions. In. an unpredictable transaction, a geeat. many processes may be 
sent lock requests, and later lock releases and not participate in, performing, the transaction. 
Using the scheme for forwarding lock requeste described. above, most-of: theae requests will 
"Not be delivered untit the transaction has been completed, and will await forwarding at-some 
level of the: Wkierarchy. Fhus while an unpredictable transaction may send outa great many 
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locks, few will actuatly be received. When the transaction is conipleted, however, lock release 
7 messages will be'sent otit for all of the participants in the transaction. ‘Because these are sent 
* out as back door messages, the processes forwarding the lock reléase meisages‘will attempt to 
combine them with the lock: requests stiff awaiting forwardiiig. “When a tock request is. 
combined with a Jock release, it is known that the lock ts unneceisary and both messages can 
‘be discarded: 


“ * Using this implementation, it is Hkely that most fock requests will be retained at a 
high level ini the hierarchy: Most'of the umriecesaary fock ‘requests Will be'canceled at's high 
"evel; before much effoit: has been expended in detvering then to their destinations. This | 
\ implementation makes it practical to run transactions that are véry uncertain - sand must lock a 
’ large number of managers but in fact perform very few:accksses. If, as assumed throughout 
"this thesis, most Of the transactions involve managers with combi Baient at stow levet of 


_~ the hierarchy, then running a transaction’ that ‘dets many ‘unmetessary: locks interferes very 


little with the execution of most of the transactions, as the lock’ requests that are hot ‘needed 
_hever reach the level in the hierarchy at which they would interfere with the more frequent 


transactions. 
4.5.4 sai the eat 


Several considerations should guide the choice of a . synchronization hierarchy for a 
distributed information system. The hiegarchy. should reflect, the patterns of locality of 
reference in the expected transactions. There are frequently natural boundaries of the 
application, such as the focal and regional offices of an ofganization using a distributed 


"information. systerh for inventory control, which can guide this choice. _ 


A second point. in. gutting. the sheice of hierarchy 
“The hierarchy sar en. r reins, ee unde 
_ allows riueh, Fanre. affician fernenatios 
communication. required. whe plararchy. exeetly. comity 
required for Syase. preci & finieniand. 


| In many cases, the tipology of the communication network closely parallels the 
patterns of locality of reference. This is because it makes. sopue. for the sites that must 
| sly coqnected, or 09 abe. connented, to. a shared 
mn) Sacco: in achive ten: smm ake semested 


| Pwo poke 1A Betwork, is nat likely, to 
reser the personaly of rnc na , | 


A third factor in the choice of the hierarchy is the «pacity and reliability individual 
sites. Given some reasonable approximations for the expected transactions, one can estimate 
the volume and importance of the metiage traffic through each site, . These should, be used 
| in evaluating whether or hot a paricubr ernie ts suitable, by insuring that each site 
has sufficient capacity tb falidle the expected ‘ieibage traffic, ‘aad ‘that Very cecahats 
yd Bhi the itilabiy ofa sh ant i utrehabte 


FET 


transactions do riot | 


‘Another factor to be considered . is the desire. to 0 aie | tecking. Lacking is 


undesirable both because it increases the number of | messages that must. be sent (the lock 


request messages), and violates the goals of autonomy and puttinl operability. A process that 
has received a lock request is dependent on other processes to a a the transaction and 
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release the lock before it can continue processing other transactions. In the next Chapter, I 
will present a mechanism that provides a solution’ to this problen, allowing a process that 
‘has feceived a tock request to continue processing other trantuctions before the outcome of 
the transaction sending the lock is known. It is stil desirable, however, to reduce locking, 
and to choose the ‘hierarchy so that frequent transactions do not’ ieee ini a ‘and 
processes managing frequently used data are rarely focked. : 


4.6 A Rejected Alternative Solution . 


This solution is of course only one of many that could be used for the problem of 
controlling transactions.. There are several solutions that provide correct synchronization 
with simpler protocols. ‘In this section, I discuss Briefly one of these alternatives aind the 


" Feasons for its rejection. micas 


Considerable complexity is introduced into the scheme by the ability to begin a 

transaction at any level of the hierarchy. If we had required all transactions to begin with a 

~ request sent to the root of the hierarchy, it wouk! be ‘easy to lock a large portion of the 
7 ” hierarchy in order to perform some transaction. ‘This could be done as follows: 


When a process receives a message, it performs whatever action is required of itself, 
_ and - passes oo the components of that message to is shildgen, a8 before. .If the process 
access, oF if one of the | 


requires, input. from. one of its children to complete its reques es 
requests forwarded cannot be completed solely based on the information in that request, the | 
"process sets 3 a lock and stops receiving new messages until it can 1 complete its requested action 
and distribute all of the necessary information to each of its children. Each process makes a 
local decision about locking and there is no difficulty detecting when a transaction has been — 


completed. 
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Figure 4.5 
Concurrency Restrictions Due te: Hierarchical Structure 
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a transactions can be performed witha locking. The hierarchy can be arranged so that the 


transactions expected to be most freqlent ao nek ‘hépire locking. Without locking, the — 
"problem of deadlock detection and preven and the “distributed atomic update problem” 
' described in the next chapter, do not arise. 


A second point is that deadiack is impossible in this scheme. The locks are set in 
| Messages: distributed jn an atm beondcast Tee if any tock set by a transaction T; 
"precedes a tock set by # frees on Ty ao the aks : 1 F109 prvnte sso one 
Ty will be able to comple jithout deadlock. : 


Another paint was iWasorated by Figo ‘When locking Ib required, frequently the 
setting of locks can be delayed, redycing the time jnerval in which ieemp are tacked. The 
scheme presented does this by delaying ordering decisions, and distributing the decision of a 


forming transaction achieves the goal of partial 
ching, 3 transs th a wn bs performed as long as all of 
| dhe processes and corpunication ln that le on th pats bowen the proces that must 
communicate in performing the transaction are functional. While this does not completely 
achieve the goal, as it is possible that two processes witl be prevented from performing some 
transaction because of the unavailability of their parent in the hierarchy, the hierarchy can 
be tailored to make this circumstance unlikely. | | | 
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Locking introduces the possibility that a process will be prevented from performing 
- focal operations because a failure has delayed the transaction setting a lock. In the next 


chapter, I present a mechanism to deal specifically with this problem. 


This chapter introduced a method for analyzing the patterns of the accesses 
_ performed by a transactions, namely transaction graphs and activity graphs. Using these 
transaction graphs, we demonstrated that sequencing of messages between processes is not 
itself sufficient to provide synchronization for some sets of concurrent transactions. Three 
classes of transactions were discussed. Many of the transactions that we expect to be 
' performed in information system fall into either the class of transactions with independent 
components, or the.class of transactions with predictable data flow. (The transactions in the 
example system discussed in chapter 6 are nearly all in the first class.) These are the simplest 


transactions to synchronize. 


A mechanism was presented to coordinate concurrent transactions using the atomic 
broadcasting mechanism developed in Chapter 3. This mechanism correctly synchronizes — 
transactions of all three classes, but works most efficiently (in terms of the number of 
‘messages needed) on transactions in the first two classes. The mechanism can be optimized 
| _ to perform those transactions that are known to be important at the time of the design of the 


system. 


The implementation of this mechanism was considered to show how the messages are 
generated from a description of the transaction, and how the processes are implemented. 
This implementation demonstrated techniques to reduce the amount of overhead caused by 
transactions requiring locking. Finally, the important properties of this solution were 


summarized. 
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ee 
Chapters — 
Polyvalues: A Mechanism for ee Atomic: Mpantes to 
Distributed Data 


In this chapter, I consider the implications of using locking on the problem of 


‘achieving the goal of partial operability. First, 'show that’ no system that uses locking can 
~ achieve this goal. A mechariléin is preséhted that solves this problem, by allowing a process 
that is participating in a transaction and thas seta tock to ‘install the results of that 


transaction conditionally, so that it can release the tock ‘and continue processing other 
transactions before knowing whether or not the transaction settigg the lock. will be completed. 


5.1 Motivation (The Trouble ‘with Locking) 


In the previous. chapter, it was demonstrated that.some form. of locking is necessary 
for synchronizing certain groups. of transactions. - Unfortunately, locking compromises the 
goal of partial operability, a3 a site that has received a lock cannot, perform local transactions 


‘conflicting with that lock until the locking transaction is-completed. One could imagine a 
- solution to this problem in which a site that has receiyed a lock could abandon that lock, 


aborting the transaction setting that lock. This must be done in such a way that if a lock is 
abandoned, all of the sites participating in the transaction Which set that lock will decide to 


eh 


abort that transaction. 


as achieve the goal of partial operability, each site must be able to decide whether | 
or not to complete the transaction without consulting other sites. In this chapter, I refer to 


the decision of whether. or not a transaction has been completed as the outcome of the 
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In order to achieve the goal of partial. operability, the locking must..mot exclude 
transactions local to procass. X..or to Y indefinitely... Failure. of Mor of ¥ or ofthe 
communication. network: connecting them may, however, delay any message sent between the 
two indefinitely.. This means that each process must at.anyipeint be-able:to decide. whether 
or not to abort the. transaction in progress without communication with other processes... 


A protocol of message exchanges between x and y that decides the outcome of a 
__. transaction can be viewed as series of process steps in each process. Each of these steps is 
triggered. by a message that may. be delayed indefinitely, so: that after.each atep, each process — 
must be prepared to decide whether oF not to.abopt the transaction. ‘This.decision. must be 
based. only on the information that that process had before beginning the-protecol andthe 
information gained from messages. received. while performing.she protecol...Both processes 
__rmust make the same decision at any point inthe protocol 


If a failure delays messages after the first step of the protocol is performed in each of 
the processes, at least one of the processes must.decide to abort.the. transaction, This is true — 
because the transaction being performed requires locking, and a transaction requiring 
locking cannot be performed with a single process step in each process. Therefore, after each 
process has performed one ‘step of the protocol, at. least one ‘of the processes must have 


insufficient information to complete the transaction, and thus must decide to abort. 


If the execution of the protocol is not delayed. by a failure,.a step must be reached 
after which one process would decide to complete the transaction if the. next. message. in the 
protocol were delayed by a failure. Let the first step of either process after which that 
process decides to complete be known as the commit point of the transaction and assume 
that it is a step of process X. After the commit point, X would decide to complete the 


transaction if a failure delayed the completion of the protocol. 
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_ Now consider che decision made by proces Y if & failure were to prevent 
step of Y, Y cannot be-effeetad by the completion of that step, ahd Hehe Must make the same 
decision before the commit point as after. This is would be a contradiction, as ¥ must either 

decide to complete before: the commit point, violnting: the iastemprion that the commit point 
was the first step after which either process decided to complete, or Y must decide to abort 
"after the commit point, resulting in an inconsistent decision. = 


This argument spplies to any umber of processes ‘attempting to perform some 
transaction requiring locking, and shows that there ts ne Way to’ mthieve the goal of partial 
| property of the ‘process mode? that the iimineditate effects "of: performing’ i (protess step are 
limited ta one process and that the observation Of ‘the ‘contpittion “of 2 Process step by any 
other process may be delayed indefinitely. 


'B.L2 Approaches te the Protilem of Abortable Locking 


There are several approaches that can be used to reduce the probability that a 
failure during the. execution of a eeiaaction requiring locking will cause indefinite defay. 
These approaches provide only a partial solution to the problem of achieving the goal of 
partial operability because a failure or combination of failures during the execution of a 
transaction can cause indefinive delay of transactions ‘that are completely focal to a 
functioning site, or cause the transaction n to be performed inconsistently, = 
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_ 5.1.21 Accepting Inconsistency 


| ‘One possible ‘solution that has not been extensively used if to aceept a small 
" probability that a transaction requiring locking will notbe' pérforitied atomicafiy: if a faiture 
occurs at the wrong time. This approach is hot appropriate for sift épplications, as strange, 
inconsistent ‘resutts may occur. If-the consequences a of not “belt “able to perform some 
: transaction promptly are worse than the consequences of a syrichPonization ‘error, (is ‘would 


be the ¢ case for a traniaction controftitig the faitding of an‘ airplane), then it may dé desirable 


to use a protocol in which a failuré'at the wrotig: tite cause w ‘transaction ‘to’ be: pattiafly 
performed, or may cause. the transaction to be incortectly sequenced’ with “other transactions. 
This s kind of eee has been used in pouanen: LaRue data base’ias. 
kriowledge, there are'tio distributed data managethént syitems that use this appreach. 


B22 Areitiee Locking 


Another approach ‘is to use synchronization protocols nat minimize the need for 
| locking. The protocols presented in Chapter 4 of this thesis and those used by | the SDD-1 
_ lstributed data bayesystem[Beritein77] are two example, of ths approach, In Chapter 4,1 
examined the problem of organizing the data base s0 as to reduce the amount of locking 
required, Locking cannot be avoided entirety, however: iintéss the'data base is réplicated so 
“that each sité has a complete copy. Such replication elitninates lockirig, ‘but makes all 
transactions that update the data base require the gies of ‘the sites, eliminating 


transactions that are local to one site. 


_ does not yet. know the outcome. If a failure delays the con 
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5.1.2.3 Minimizing the Windew of Vulnerability 


The approach most frequently.taken.to lacking in. a distriljaped. system ia to minimize 

the time. interval during..which a. failure causes indefinite delay....One example.of this — 
approach is the two-phase conut .protol. deusibed. hy Gray [Gray7/l, Each. site 

_ participating in a transaction goes thrangh,two,phases, a lock. phase. in, which Jocks are set 

“and the site.computes the results of the transaction,.and.s. wait, phase during. which the site 

to make the updates. requested, hy the, transaction without, farther, input, from. other sites, but 

| | sien of, the Jock phase at a. site 

the site can decide. on, its, own.to abort the transaction, and. all sitea.will eventually decide-on 

"their own to abort, or pe tald.pf the decision,.te abort, 1a talhure delays eoesmages during the 

| wait phase, however, a, site must walt sunt). it receives, a menage pas gutcome of 

o transaction. 


B. necessary 


Figure 6.1 gives a finite state machine description of the action of one of the sites in 


Grad: 


begin this transition, and the meas ‘tent in —_~ this ‘tranaition' an ‘italic oe any 


In the lock state, a site waits 3 for MeSEAZES ce intornanen essay 
complete its portion of the transaction hy determining the new yalues for the ‘tea, at that 
Site updated bythe transaction, After these have been received, the. site enters. the, wait 
phase and sends an acknowledgement message indicating this fect If a failure delays. the 
reception of messages by a site in the lock phase, that site can abort the transaction by 


sending an abort message and entering the abort state, discarding any computation done by 
the transaction. In either the abort or the done states, the site is ready to accept new 
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transactions. The acknowledgement and abort messages sent by: the sites, are-accumulated by 
a coordinator for the transaction until either all sites Baral ompented MLnoenees:: or any 
site has sent an abort. The coordinator then vigeeinonts ons or abort mesmages 4 all of the 


participants. - 


‘The oe behind this protocol is thas the me that sack.site spends during its 

lock phase computing the results of the transaction is likely to be longer than the time spent 
; sonot#ad tdi cD ab ob he 

during the wait phase. This is not necessarily true, as one site may take much longer than 


Sek S, 


Figure! 51. 
A Two-Phase.Commit. Pretooal, 
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Lampson and Sturgis (Lampson76) present another commit protocol that includes an 
extra round of message exchanges to avoid this problem. In their protocol, no site enters its 


wait phase until Ee ee teen errs ere omnpleted’ at Ait sites. 


5. 1.2.4 The c Polyvaie Appr 


The motivation behind preventing a transaction ibiat holding’ on te a tock 
indefinitely is to be able to run other transactions that need t0 access the data that has been 
locked without indefinite delays. Frequently, the resuks produced by a transaction depend 
only loosely on the input Vata seen by that transaction. ° IY the dutputs to be produced by a 
transaction holding a iia are known but the outcome of the transaction is uncertain, there 


are two possible sets of current values for the updated items. One could use these two sets of 
values to determine fer sume transaction waiting to access the updated values, whether or not 
that transaction depends on which of the possible sets of values is correct, Any transaction 
that does not depend on which set of values are used can be run using either set before the 
outcome of the transaction with the tock Is decided. The potyvalue scheme described in the 


next section is a generstization - this idea. 


5.2 The Polyvalue Mechanism for Avoiding Deley Due to Locking 


This section Presents a mechanism that in many cases solves the problem of insuring 
that no transaction is delayed indefinitely due to a lock set by some othe transaction. 
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5.2.1 The Polyvaiue Concept 


If a two-phase commit protocol is used to perform a ‘transaction, a site that has 
‘reached the wait phase snows output values of the transaction. If those values could 
somencw be conditionally installed, such that a transaction accessing one of the updated 

items would see both values, then the locks on the upiiaed items could be released. This can 
: be accomplished ‘by installing what I refer to as a polvvalue for each updated tem. A 
polyvalue isa bookkeeping tool for. keeping track of several potential current values for 3 an 
item, depending on the outcome of currently pending transactions. 


A polyvalue is a set of pairs, mdae where v ‘is a value and c is aco condition, which is 
a predicate on a set of identifiers for transactions. The palr < “ve ‘in a polyvalue for some 
item I specifies that I has value v whenever c is true when c ts evaluated ina model where 
transaction identifier T is true if T has been completed. The conditions in a ainigle 
polyvalue must be disjoint (no assignment of truth to the transaction identifiers makes two 
conditions in the same polyvalue true) and complete ran any assignment of truth values, one 


condition is true). 


Each transaction is assigned a unique transaction identifier. When a site that has 
“reached the wait phase for a particular transaction T cannet-determine quickly whether. T 
- will be completed or aborted, that-site installs polyvatues for aft of the iterna:that T is trying 

to update. The polyvalue installed for an item I has two pairs,:<v’', Pacand <v,-T>, where v 
was the value of I before the execution of T, and v’ is i valve, produced by 'T. This 
polyvalue describes the possible values that could be the current value of I, ere on 


the denial outcome T. 
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‘Before instillation, exch :polyvatue.is simplified .in three-seaps., Fleet, individual pairs 
are expanded. Any pair vo» where -w to cheailt = polynaion: cmcaunes by a: rou: -of pairs. 
This group: containe sane pair-of the:form: Le her: for each, parce that was: in v. Next, 
| redundant pairs are-coneeed. ‘The:pairs seeped. <gter share he at Np are replaced iby 
the single pair songs Mage. "Phase sedundant pairs can arise because iets poubie that ‘several 
different ‘possibie: “outcomes of the pending transactions cou: produce the: same value for an 
item. Finally, ‘the condition-attached. to each: pale arp, and any pair <1» for which 
c is logically false is:diacarded. | 


. This simplification. procedure reduces the petyeahoe constrectet to one in which each 
pair has a simple value, and the number of pairs Js minimized, A polyvatue with a single 
Pair <V,C>, aust have a condition c which i logiaty tra, and is indistinguishable from a 
simple value. Thus the ‘Procedure for constructing polyvalies for the results of a pending 
. transaction can be described without trating the cuss where the new or old values of the 
wee items. are thereat pointes as special ian. my. . 


5.2.2 Performing Tramsactions on Polyvalues 


oA: {transaction ‘epecatiag on. potywalued pathos hncomnas. pe saci 
_palytransaction a pic ee sc ag ed si a ik cama walue- 
Prociuced under any ‘ponible outeome of patding. trneactions: ‘Te Stings =: smple 
- siscussion f:polyteancactions | 


I wit first describe the computation phase fs polyranacion, in which input values 
are read and outputs are computed. Each polytransaction T comes of 8 set of ahernaive 


transactions Teach .of which performs the same transaction on a different. set values for 
input items. Each alternative transaction T, is tagged with a condition c, which is derived 


ER SIVAT RPT RUPE ITE ST: SEER TE ed: 


- 135 - 
from the conditions on the input values read by T,. Each. polytransaction begins with a 
_ single. alternative transaction T,,,,, which begins to. access items, in- performing . she 
_ transaction. When an alternative transaction T, accesses an. item, whose current. value, is a 
polyvalue v = {<v,cj>}, T, is partitioned into.a. sat. of akernative transactions, AT cpg} each 
of which has the same history as T,, and each of whith, accesses, one value ¥, from ¥.and 
acquires the corresponding condition c;, in addition to the previous condition, ¢, on To If 


cAc; is Perel false, then T, Ac, a" be abandoned, and not ae 


Thus the number of steernasiv cnet res polytransaction T is run. 
_ Each of these aiternative: transaction’ runs up to the wait phiise’ (1: each runs uritit the 
outputs have been computed and distributed to all of the appropriate sites). Each site 
receiving outputs of T constructs a polyvalue for _. item I to be updated. This polyvalue 
contains the pairs <v,c> where v , is the value produced by T%; for he Se 


. if all alternative transactions of T produce outputs for some item I, then this set of 
> pairs will be complete and disjoint If, however, there are some alternatives of T which do 

- not prodiées a value for I, then the conditions of the ‘aiternatives which do produce values 
for I ‘will not be complete. This can happen if the decision df whether or: not T updates I 
depends on the input values seen by T. Under any outcome of pending transactions for 
which T will not produce a new ‘value for 1,1 would retain its 8 previous value. Therefore, if 


the conditions on the alternatives of T which produce a a new value for I do'not form a 


1. “As wilt be shown, outputs produced by W'anernative transaction with a condition that is 
logically false will never be used. 

2. T begins with a single akernative with condition true. AS the computation phase of T 

_ progresses, akernatives-of F are parttionad acenrding- ao: the conditions: onthe polyvaiues 

that they access. Because the conditions on the pairs.of any. individual_polyvalue are 

complete and disjoint tt the conditions on the alternatives of T are: rat eaey ak a and 

disjoint. _ ; ; it 
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complete set, snother pair ev'mx'> is added where v’ fs the:previous value of 1, and ¢ is the 
Sere ee On ee eee oe Se ee eee The wait phase of a 
_ polytransaction provestls at described ubove. Should a jication failure interfere with 
the wait phave, = prac be rfc Pom te ep ee eee ad he 
ae = 


5.2.8-A Simple Example 


Let us consider a simple example involving three hems at three sites, and three 
transactions on those items. Let.A, B, and C be the items, and let. the.transactione be: 


Ty +184 > ROthen [A = A- 100, B - B + 100} 
T 2 = IF B 2 100 then {B= B - 100; C = C + 00} 


Ty if B > then B = 10508 


Now assume that before the transactions are run, each.item has value fu. If a failure occurs 

during the wait phase of T, preventing the site holding. B from Jearning the outcome of 7 ;, 
“then that site gives B a polyvalue of {2007 p, 400-7}. If Tz is now run, it will be run 
: _ as a polytransaction, because of the polyvalue of B. 72 would produce new values for B and 
C of {<100,7 >, -Tp} and 200. If a fatlure occurs during the wait phase of Ty again. 
preventing the site holding B from learning the outcome of T'g, then. after simplification, B 
receives a polyvatue of {<O,77 AT 9>, <l00(T AT 9)W(AT JA“T-p)>, <200,1 ;A-“T 27>}. Now, if 
T 3 is run, it is performed as three alternative transactions, Two of these. alternative 
transactions produce updated values for B, while the akernative for aT AT g does fot, 
because the input value for B read by that ahernative transection iP toe stall. Thus the 


1. One could alternatively atways add this pair, and rely on the simplification procedure to 
discover that ~<’ is logically false when the other conditions are complete. 
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_polyvalue assigned to B by F3 after simplification is) {<0,77,AT 2>, 
<105(T pAT 2M-r 1 AnTF 2) <210,7 ;AWT 2?}- 


This "example shows the mechanics of ‘manipulating ‘polyvalues, to perform 
transactions, even after the occurrence of improbable failures. "From this example, it is hard 
agitoce 


to see what has been gained, as one cannot determine from Inspection what the values in the 


data base are, or what transactions have been completed: 


The answer is that in many cases, a polytransaction will produce ‘simple output 

. values. This is true of many query transactions, which attempt to determine whether or not 

the value of some item falls.in.a certain range. In many.cases, a query about an item can be 

| answered without knowing the exact value of that item. A Lasdiaae! can provete all of the 
information necessary to answer common queries. Consider, for example the tes mage by T2 
on B. The decision made by this t test is the same e when speed to both » component sed the 
polyvaiue for B. 


Another area where polyvalues are useful is that of transactions that have real world 
‘effects, such as authorizing transfers of money, or atocaung a real work TESTOR, like a seat 
on.an airplane. For such transactions, it is Rroqoenty. more Important to know what the real 


world effect is than to know what the eventual. values in the data base are. If such a 


|. transaction is run on an input set containing potyvalues, then the real world effect can be 


accurately determined when all alternatives produce the same effect. In many applications 
znporant real world effects can be Crermwied without knowing the exact values in the 


database: 
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Consider, for example,:2 transaction. which 1s te withdraw: fands from a: savings 
account for which the oocrent balance is represerted'by a polyvalue,. The important effect 
that the transaction crust decile quickly is whether er not the cuatorner ts bu receive the cash 
from the withdrawal. Computing exactly the new balance in the account need. not occur 
rapidly. The transfer of funds depends only loosely on the belance in the account in that it 
need only be determined that that balance 1, under all peuble outcomes of pending 
transactions, greater than the amount withdrawn. Thus in most cases the withdrawal can be 
mick authorized. 


6. Recovery of Pending Transitions 


The mechanism described above inalspoivaies forthe resuks of a transaction T 
| delayed in the wait phase by = temporary failure. When that failure is recovered, the wait 
phase of T can be completed, determining whether T is to be completed or aborted. Thus 
the value of the transaction identifier for T appearing in conditions in the pairs of 
_ polyvalues can then be determined. ts ae 


A site Yearning of the completion or abortion of « transaction T can reduce its 
polyvalues by. re-evaluating any condition that depends on the outcome of T, substituting 
either true or false for T depending on whether T was completed or aborted. This 
"substitution simplifies conditions that invoived T, and upon siemplification, some of these 
“conditions may become logically fale Thus knowledge of the completion or abortion of 
pending transactions can be used to pion the number of possible values which a polyyalve: | 
represents. Eventually, if the outcome of all pending transactions is known, each polyvalue 
will have only one pair with a condition that is not logically false, and thus can be reduced _ 
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to a simple value. Some mechanism must be provided, however, to propagate the knowledge 
of the outcome of a transaction 7 to sites holding polyvalues with conditions involving T. 


Such a mechani must insure that all sites that hold a  polyvalu with a condition 
dependent on a transaction T will eventually Far of the outcome of T. We also desire that 
knowledge of T be deleted when it is no longer necessary (Le. when no condition involves T). 
_ The record of the completion or abortion of a pending transaction is similar toa commit 
record (Reed 78] for that transaction. Unlike a commit record, however, knowledge of the 
outcome of a transaction may still be needed even ‘after af of the output values of the 
. transaction have been installed. Any polyvatue could potentially refer to any pending 


transaction. 


One could have each site maintain a abe of outcomes of pending transactions, and 
use a system-wide garbage collection strategy to delete entre that are no longer relevant. - 
While this scheme would work, it would be inefficient in the case that dependence on the 
outcome of pending transactions does not in genera spread very far. Most sites do not need 


to nhow the outcome of most pending transactions, 


Another possible mechanism is to give a site that creates a poyreee for a pending 
transaction the eesponsiolty of Maintaining a record, of tg outcome ef that transaction until 
es such a record is no longer necessary. When a , site wishes to reduce a. polyvaiue, it must ask 
all of the sites that are responsible for. maintaining a record of the outcome of the 
transactions appearing in that polyyalue of those outcomes. To do so, 40, ‘information must be . 
passed along with the polyvalue to determine the relevant sites to ask. This scheme is 


similar to that used with possibilities by Reed (Reed78]. 
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There are twp amin geobtems with using this scheme far keeping track of pending 
tramsactions. Cine qeubiieen ts whut d would tbe aidfficstt to xietessaine when the record of the — 
outcome of a pending transntion is no longer needed, and seme form of garbage collection 
may be ‘necessary. ‘A seeonl problem és that the amenages sent to inquire about the outcome 
of a transaction may spllece a ‘burden on the communication vant, as the inquiring 
message may be sent many times fence for each attempted access to the patyvalue) before the 
. outcome wf the transaction 4s determined. The scheme described below overcomes these 
problems by distributing the responsibility far maintaining the outcome ‘of a pending 
_ transaction among Senn it tere polyoioes pent on she suman. 


Each site maintains a ‘table, referred 0 there is ‘the pelpralue table, listing the items 
that it holds that currently ‘have polyvalues. This ‘table ds-used ‘to focate all of the polyvalues 
_ that can be reduced when the ‘site reccives a menage indicating. the ‘outcome of some 

pending tt transaction. A second table ‘maintained ‘at each site, known as the transaction table, 
keeps track. of the sped of knowledge of pending transactions, Each entry of the 
“transaction table contains a transaction se, its outcome, (completed, aborted, or | 
pending), and a dist of sites ‘to which this site thas. sent dnformation dependent on the outcome — 


of that transaction. 


To maintain dts ‘transaction table, a site must make an entry for each transaction 
identifier that appears in a condition of a polyvalue at the time that that polyvatue is 
installed! When aa site sendsa message containing a polyvalue to some other site, it must 


1. No action is required if the site already has a table entry for that transaction. 
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record the name of the site to which the polyvalue was-sent in the transaction table entry for 
each transaction identifier that appears in a condition mene: 


The information in the transaction abe in the various sites is used to control the 
distribution of knowledge of transaction outcomes. Each site that receives a commit or an 
abort message for a transaction that it previously knew as pending can update its table entry 
for that transaction, and reduce any polyvalues that depended on that transaction. A site is 
responsible for informing all of the sites that are listed in its transaction table entry for the 
transaction of the outcome. This list was constructed to include all of, the sites that. were 
given. information dependent on the outcome of the transaction, and therefore may hold 
polyvalues dependent on that outcome. Once all of these sites have bee informed, the table 


entry for the transaction can be deleted. 


With this scheme, knowledge of a pending fransaction Propagates only. to those sites 
which have received polyvalues dependent on,.the outcome of that transaction. If a great 
deal of computation has been based on the outputs of a pending transaction, then. informing 
ail of the appropriate sites of the outcome of that, transaction may require many message 
exchanges! ; If the outputs of a pending transaction are not used, however, only the. sites that 
hold those outputs need be informed of the outcome of the transaction. | | 


Figure 5.2 shows how this scheme works in the. example described above. Let T, 
and T: be the two transactions described earlier on items A, B, ‘and C. Assume that these 


items are held by sites A, B, and C bas ‘The figure a shows the values of these items 


1. In fact, if the polyvalues depending on a pending transaction are ‘used frequently, a site 
“may have to be informied of the outcome of that transaction déveral'tfines. “it is possible for a 
' site to receive a polyvalue dependent on the. outcome of a trat after that site had been 
informed of the outcome of that transaction and had forgonen t that outcome. A site does not 
need to remember transaction outcomes indefinitely. ah as 


= 2 - 
and the tables. of pending transactions in the sites maintaining these items at several sages: _ 

initially; after T, is suspended; after Ty te suspended after T, is eventually completed; and 
after To is eventualty aborted. | . | 


_5.4 Use of Polyvalues in the Hierarchical Locking Scheme 


The discussion of potyvatues thus far has been at a relatively high level, so as to be 
applicable to any distributed system in which locking without unbounded ounded delay is needed. 
“The aden amen described above could tence scle oats into _ of the 


‘apply these ideas specitcaly ‘to the distributed tek ‘scheme described in the frevious 


chapter. 


Reval that inthe focking scheme of the previous chapter, any process producing 
outputs to a transaction depending on inputs obtained from another process which is not one 
of its-ancestors ‘In the hierarchy is sent < tack requiét méssagé. The lock request inesaage 
catises the protess to refuise to receive any new riettages pertaining to other transactions viitil 
the transaction issuing’ the fock is completed: The prodeieel Invetved in the transaction 
exchange messages until each locked process has sufficterit information to produce its outputs 
and release its lock. In order to apply the concept 0 of polyvatues, this locking strategy must 
_ be modified so that each locked process goes through two phases, a computing phase in 
which the process could abandon its lock and cause the transaction. to be aborted, and a wait 
phase in which the lock cannot be abandoned, but the output values are known. 


I will ‘first consider the case of a predictable transaction, where the Sune | 
simplifies the task of deciding when a transaction can be completed, as ‘each process making 
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Recovery of Pending Transactions — 
Initial State 


a | a Bhs - 
100 00 100 
Transaction Tables: 


(empty) _— (empty)... (empty) 


“After T pis Suspended 
{<0,7 )>, <100,-7 )>} {<200,7 >, <100,-T Pp} — : 100 
‘Transaction Tables: | 


Typending{} = ypending.h. =... (empty) 


After T is Suspended _ 


A 6 erates 2: av gely oR qwere re} 
{OT 4007p} {OAT AT p>, QT AT >, {<1 9>, <200,7 2>} 
Transaction Tabless 0) 


Tppending{} --s—s—s—sts«*'ypending fC} pending.) 
ie ge Bie eae he Tei ae Topending, G2. Cts poets ne ae aad 
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| After Ts has been completed, and A and B have been notified. 


rm biden’ waif obinein a oat! . 

‘ fOr: 200-7} {<100,7 2>, <200,7 2>} 
Transaction Tables: : | og 

(empty) | T j#lone,{C} | | | Topenting 
After C has been notified of T 
And T'y has been aborted. 

BL. "sy 6 

o. oe ae ® ae 
Transaction Tables:. =f 


_ Updates knows when i can complete those epdates, ad the set of pcs making updates 


is known in advance. 


Pas im Geceisiaattioks an see bac of sransection coordinator. 
The transaction coordinator has the responsibility for determining when all of the processes 
jevewved in the transaction Ihave reached the walt phase. To begin the transaction, messages 
must be sent to each of the proses tneelved din tiertransaction in a single atomic broadcast. 
The protocol of Chapter 4 must be slightly modified to send every goousss,that.is.to. perform 
an update a lock request message, Recail that the protocol of Chapter 4 sends lock requests 
only to those managers that cannot sovnplen the transaction in one process step. The extra 
locking is needed in implementing potyvalues because we wish to be able to abort the 
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transaction if the completion of the transaction is ‘delayed, and thus cannot ‘allow any 
manager paces in the transaction to complete its portion of the transaction before the 


' Each’ process performing an update thus receives a Jock equa along with any other 
" instructions: for completing the transaction. ‘When a process ‘has ‘eiough ‘information to 
| perform its ‘update, it'sends a "ready" message to the coordinator: “(For any process whose 
update can be made ‘without inputs from other processes, this ‘happens immediately). Before 
__ sending the "ready" a, process can-decide, to abandon its Jock at.any point and cause the 
transaction to be aborted. “After sending she “reagy” message a. papas enters its wait phase 
_ and. cannet, abandon its lock, When all of the. proceages that , were.sent lock requests. have 
answered "ready", the coordinator decides to complete the transaction and, sends | “complete” 

messages to the back door sabe of the processes which received locks. Upon recall of the 
~~ “Complete” message, a process completes its update and enables 3 reception of new requests. If 
* too much’'time elapses béfore the coordiriator ‘receives “rhady" messages from all locked 
processes, the coordinator ‘can abort the update ‘by sending *ybort” messages to all. “The 
7 "ready" “complete”, and “abort” messages must all be identified with a unique identifier for 
‘the transaction (probably assigned ‘by the trifisaction ici initiated! the transaction), 


Ped St 


so that uaa’ maisige d6 not cause confusion. ” 


Each process in this akc goes through brie phases, a lock. phase, before. sending 
the ay MEAS and a wait phase after sending that message. After having sent a “ready” 
message, a process ‘knows the new vaiues that some items in its local state will take on asa 
“result of completing the update. The process ‘can, ‘instead ‘of waiting £ for a “complete” or 


“*abort" message, decide to install polyvalues for these tere. Each “data manager process acts 
‘like a site in the polyvalue scheme described in the first part of this ‘chapter. Messages sent 


eras MET Ra 
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from one process to another containing data items or results compute 
contain polyvalues as well. 


from data items can 


Two problems must be overcome in extending this scheme to atbitrary transactions. 
First, the coordinator must be able to know when the transaction can be completed, as the set 
of processes making updates is not known in advance. ‘Second, each process that participates 
in the transaction must be able to determine when it has recelyed all of the messages that it 
will receive as a part of the transaction, 20, fant it knaeve sheen 0 Panter She wt phase 


The at ft protien we dct Capt compen Weight 
ntraduos inditor can walle ‘act aé the 


rettas 


We can modify the completion weigh scheme to alow unceriain ringtone 
performed with a two-phase Protocol. Each process step of an pncertain transaction which 
Prepares 2 set of output values tobe insaled must return some completion weight tn, the 
coordinator whether o7 not it also sends messages | to other. processes, - . The. coordinator . thus 
receives messages containing completion weight from pach process that has updated Atems to 
be. instafied. When the completion weight sent. to the coordinator. “reaches one, the. 


. “coordinator sends out leck-release messages as before. These ae gas are 
: Gistributed as described in Chapter 4. 3 RS ae AE ga tas 


In this protocol, cach manager can at any pin decide to, abandon is, lock and es 


continue Processing other transactions. _To do 40, a mreneger raat any updates that the . 
transaction has made as potyvaiues, and simply ignores, any further peng about that 
transaction (except for the feck coe or abort —— from the Sparuaction coordinator). 


-.. Manager abandoning..the transaction is not needed .to. cag 
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This action may or may not cause a transaction to abort, ‘depending on whether or not that 


transaction. requires. further participation by the, manager, which has,abandoned it. If the 


fete. the transaction, then 


eventually,. the. completion. weight returned to the .c or will..sum to 1, assuming no 


other, manager decides.to abort, If, however,. the. manager deciding to abandon, the 


_ transaction must.perform additional processing, to. complete. the transaction. (either by 
_ Supplying more. inputs,or making. updates, the transaction .yill..not. complete, because the 
_ .portion of the transaction dependent on the, abandoning .yqnager can not be completed. 
i Eventually, the, coordinator will decide to abort.the transaction. . 


". This scheme allows the polyvalue mechanism to be applied to the execution of 


“uncertain transactions. ‘In this scheme, each update made by the transaction goes through 


two phases, a lock phase before it is computed, and a wait phase after it has been compirted, 


. and the manager. holding the updated item has replied. ta.the goordinator. 


“Another point that should be noted about the use of polyvalies in the locking 
scheme of” ‘Chapter 4 is that the protocols that allow abortable locking described above may 
require that’ more lock requests be sent than the simple protocols of Chapter 4. ~ Note, 
however, that any transaction that does not ‘require locking with’ the simple protocols still 


does not require locking, we are only increasing the number of locks sent. for. transactions 


that already require locking. 
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_ My Restricting the Spread of Pelyvalues 


The polyvatie mechanism is expensive in that polyetuied contin’ a great deal more 

‘Space than do simple values, and’ a polytransactioi-nily ‘Tequive °gfeat debit more 
computation. The simple analysts of thie potyviliyicheme aid lltiutition ‘of the protocol 
- reported in an appendix to thi tials deiwontrabe that im thet Uhe” expect ‘Wuniber of 
polyvatues in: a distritigted ‘information’ iyicen "ie qutie: seat’ Shield: tifther: contrat be 


necessary, any site can prevent the propaga agation of p 


See ee iin as ane nite wing i of the 
i peapia taps’ wid instead 


“a waiting until it can. reduce the seliratin These. derisions 


possibly delaying Imporvant transactions that could Md have eae perfe 
potyvabves 


In a system with real time response requiternents,, fe tain Ua asbelabte' to ‘expect that 
__ the set of transactions that must be performed in order to produce | needed results at the 
: proper time will be known, Ati precisely these tranvactiqns that should a be. performe med 
.Palytransactions, so that if possible, the needed results can be obtained despite » ancertanty in 
the database values due tothe presence of pending tranagtions and polyvaties, 


Consider a system controlling some manufactaring * operation “in ‘which several 
ikcity "ached “neat the 
components that they monitor and control. Several different. kinds of transactions act on the 


eth 


computers are used to control the manufacturing and are 


data base. There are data entry transactions that are run periodically to enter data about 
the operation being controlled into the database. There are also monitoring transactions 
which are run periodically to determine whether or not the database values indicate any 
potentially dangerous conditions requiring immediate corrective action. The monitoring 
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transactions are structured so that many examine ‘onty: values local to some ‘site in order to 


insure that a communication failure cannot interfére with monitoring. « 


In addition to these two kinds of renee there are control transactions that 

direct the completion of specific manufacturing ‘tasks. There are also transactions that 
implement administrative decisions to change the mhantatacturing process'by: modifying items. 
representing parameters to the- ‘contret and. mormoring: traneaétions, and: transactions’ that 
 attow the state“of tie manufacturing process-to be exathined: “The! monitering transéctions 
need to:be-performed in real time in-order te prevent futures iw the:physical oémponents of . 
the manufacturing ‘process or “bugs” in the:controf transactions: from ‘creating’. hasardous 
situation. These monitoring transactions examine the values produced by the data entry 
transactions and the peromee? of ae poe: to detect ahertraiea Ras! He normal set of 


yee, 


parameters and data ‘Inputs wil not rigger corrective action. 


gv Page on TEN 


| ~ In arder.to: insure poeKnnte eet cei es beta 
_. Shoukdvbe used for any date items: that might:be read by the monitoring: transactions: “The 
‘control effects. of the monitoring transactions should be independent of the enact value of the 
date items. describing the process, as long as these data items reflect normat eperation. ‘The 
“transactions whieh: direct: specific manufacturing ‘tasks: and the transactions implementing 
. administrative decisions may involve updates to data hems at.severat: sites, and thus may 
require locking. The locking performed for such transactions should allow the creation of 
polyvalues for their outpun if some failure prevents the lcs from being quickly released. 
_ Transactions representing administrative se of the manufacturing process or 
control of specific functions may be deemed less important, and may not be executed as 
polytransactions if necessary. Any process hokling items accessed by the monitoring 


| . - BO- 
transactions, however, must be prepared to install polyvale 


“56 Summary 


This chapter hes baen devoted to a discasion ef the: “diaeeibnenedttomnis:-update” 
_ problem. It was: shown. thad.it Je inapaeaiole, given the falluce semantics ef the process model, 
to construct 2 protonel. which: penforms a distaibuted update atomically while not delaying 
access. to. the updated: items indefinisely at any fenctioning site... Several.strategies were 
“discussed to avoid the: distributed: atomic tupdage paige in -rmesectiog synchronization 


| The remainder of the chapter presented a concept refered to as polyraise, which 
may provide a practical solution to this problem in ‘many casen. Polyvalues allow an update 
- to be performed conditionally, such: thet: beth the apdated::andmencapdated : values are 
presented ta srbsequent transactions: in an: informationsaystem. ‘where: the saost: Irhportant 
effects of transactions depend only loosely on the exact vakees stored;ift the date base, the 
| -polyvalue scheme allows these impostant effects to. be determined! quickty; even when: the 
exact values of items: in the data base are uncertain due to: transictions that: have been 
started but not yet completed. 


| This chapter presented some simple examples of the “mechanics of manipulating 
polyvalues and discussed a possible application of polyvalues in a process control system. 
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Chapter 6 
: Application of the Techniques. te:the. paren ofa: Distributed 
Information System 


of an overall 


The. past four chapters of this thesis chaye prevented ack pect 


approach to the” problem of Tobust synchronization ‘ini a” ed vitborrnds tion system. In 
this chapter, I present an aap! of a distributed information sven ‘nd show how the 
techniques tbat Ihave: developed. can be applied terderive synchronization scheme that 
satisfies the goals set forth in Chapter 1. This solution is enpee ‘with those using other 


distributed: ‘Syrichronization schemes. 


6k TheProblem « 


The chosen example is an inventory control system for a chain of supermarkets. The 
«problem .is adapted from an. example given in.[Berasein77) .The,date base.is used to. keep 
track of the.quantjties of various products.(cans af, beans, paper napkins, etc) gn. hand, on 
_. order, of. jr transit at each individual market and at the. wayshouses,that supply the markets. 
The supply. chain of the sapermerkata | is blerarchicals: with groups of. markets; supplied. by 
local distributers, groups of local distributers supplied by ragiogal-distributers, and.s0 forth. 

’ The followings sections describe the date ~_ the Senachont to be digpiplinee 
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6.L.1 The Data Items 


For each location (warehouse: or “superrgarket) the dats base conthine's sgt of data 


items describing each product. These are: garde kl gee ce bag El 


Sa on Hand (QOH) -- The quantity of that product stored at 
that location. 


Hand (DQOH) ~ T cal of how much of 
a to try to hae fe Brad we iaity ‘deni’ 
i: MOPS OF eR eTH FERS). AMEE. FURAN Ey 


Re-order Quantity Threshold (RQT) = At oninvinate quan ant of the : - 
product te‘heap on hand: sot aes RQFiiah orders 


_ Is submitted to bring QOH up to DQOH. — s eeeilepiiy on ee 
Quantity on Order (QOO) — The amoung of § hat me 
sync) Tm mf not yet 
‘been delivered. 


Quantity in Shipping (QJ8).-r“Flae-amount of the product that has 
been shipped from the distributer for this location, but has not yet 
been deliversd. 


The data items perteinifg to each Of the protests Sire iebeperiiertty epi y exevraniea and updated . 
(i.e. there is no single trafisaction that accessed input arts perthiting® to” two or. more 

Products), sty E will consider ‘otrly the items pertaining: tod ‘single product! mn fact, a typical 

‘supermarket may stock a total of 10/000 different products, babaahaioad untepeitent ot of these 


ats teres 


The five items are maintained for ‘eith secations Waa: or Oe es To 
distinguish between items describing different locations that are used by the same 
transaction, I will use subscripts, such as QOH, to designate the level of the distribution 
hierarchy to which an item pertains. Level 0 designates the local markets, while increasing 
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subscripts designate more global distributers: ‘Phis is sufficient to distinguish the items 
‘because each transaction accesses items pertaining’ tot most tWe lecitions: a Jecation’ and its 


~ supplier. a ee 


6.12 The siwaias : 


inuading 6 of a truck, to the data base, and to determine hen some real world action, should 


PRB Re TS sy 8g) 


be: performed. to keep supplies of all products available. For each sige there are four 
SE ee eee Polwt of Sab, Re Order Shlbpiat and lectiviniy. 


tartering ot 


Point of sale ¢ transactions (P transaction) update, the quanti on hand. to reflect a 
7 customer - purchase, . P transactions take D agin A sis on the, QOH for the locations 
corresponding. to supermarkets and not on _ those, for the distributor. For a typical 


a id there are about 23000 P tran actions pet go tee . 


_ Reorder ‘transactions (O transactions) ; generate new orders for merchandise whiten 
has. been: depleted. An O transaction éxamthailthe QOH, QOO, RQT, and DQOH for 
some location and prodyees a new value for QD0.: Fer: ench:Jacaticii, approximately 2000 Oo 
_ fransactions are performed per day to detarmine which products must be ordered. aaa, 


Shipping transactions (S transactions) reflect action’ by a distributer to fill an order. 
A shipping transaction examines the QOH of the distribagir and the QJS and QOO of one 
of its customers in order to decide how much 9f-the produst to ship to that customer. The S 
transaction updates the QOH of the aistributér’ sind-the OO of the customer to reflect the 
shipping decision. S transactions are performedt at the’ rate of about 15 per day per location. 


37 9 
oN 
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location. Each R transaction adds the amount received. to: QOH, and; subatracte it fram: QUS 
and QOO. About I5 R transactions take place for each site each day. 


| These transactions are summarized in Table 61. In the paper which is the.spurce of 

‘this example, the authors were unconcerned with the details of how each transaction derives 

"its outputs from its input values. I have therefore made some “educated gusts” is ‘daciving 
a ce cont cin ot ie mentions . 


Note in particular that the roaning. rgpmations are prewmed to take, ab 2 
| parameter the amount of the product received, and to use that amount to update the items 
_QOO, QJS, and QOH. An R transaction ‘always has independent components, because the 
new value of each of the ives updated depends only on its previous value and on the 
parameter Q, Another possible interpretation would be to use the value QJS to determine 
the amount received, thus making the new values of QOO and QOH depend on QJS. I 


| Table6i. 


Transaction | wars Dewriptidn Frequency 
Poe's QOH QOH 500 
=, | - QPO, ~~ HQOH; QO0;, BAH, RQTY 2,000 
a Gat ee QOH), = SofQI;,, a 24 ROH). e oad - 
a ROH, = QOH AQ). : | 

Q90, = QO0H(Q 
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believe that my interpretation more closely resembles what would happen in a real inventory 


control system, as the parameter Q represents the amount actyally received, and may not 


_ correspond to a for variety of reasons. 


Having a complete description of the data base and. the transactions to. be performed, 
we can now proceed to analyze the system.using the tools developed. in Chapters 4.and 5. 


62 Analysis of the Transactions. 


In this section, I present transaction graphs for the transactions to be performed by 
the inventory control system. These are analyzed tacaxplore the ways in. which the 
transactions interact with each other.. This analysis is:meedto detarminethe protocols needed 


| to perform the’ transactions using several different-orgenizations of the date base (choices of 


which items are held at each: site). The choice of the-synchugpization. network for each of 
these organizations is discussed... Finally, I discuss-the use of polyvalues in this distributed 


' information system. 


6.2.1 Transaction Graphs for this Application 


The transaction graphs for typical Eanes from aie fo classes are shown in 


Figure 61. The P transactions are the simplest, a as each P transaction accesses and updates a 


single data item. P transactions will have Independent components in ‘any organization of 


the data base, 


|‘ The:R transactions are somewhat more complex,-as they. access and update three 
different items. - As noted in the previous:section, howeves,.the new value af. each of these 


_ items depends only on its previous value. Therefore.the: transaction graph of an R 


transaction does not contain arcs interconnecting the three updated items. 
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The © transactions update 2 single dain seein (QQO fur: some Weathori) but do so 


based on several inputs. As shown tit Figure Gf: the thanindtion giaplt for a‘ transaction 


has arcs connecting QOH, ROT, DQOH, and QOO 6 QO0.’ Ae a 
“The S transactions are the mest complex. Bici?S trendsection tipdates ewo'items (Q5S 


for some location and QOH for its suppliit), baiell'o the pAbvléde valees oF three different 
items. The transaction. graphs for these transactions thus contain cycles of arcs, connecting 
QUIS, QOH, and QO to both js and QOH. ‘Fhese cyclts: indicate that S transactions are 
sd oa a en reece rel ha 


‘The four kinds of -sransactions taking place at vastous: levels. of the hierarchical 
organization of the’ focattons internct: ‘This. interaction: te iultented: by. Figure: 6.2, which 
shows a joint trensaction graph! for the traneuctions:txhing place-at three: towels of the 
hierarchy. The joint transsewen graph i+ conswacted froatithd individual transaction graphs 
“in the sume way ther a joint scebrity graph ls condmuched freme-wndbridwal aetteity. graphs. 
To distinguish between the transactions taking place at different levels: of: the: aisaribvution 
| hierarchy, each transaction identifier is given a oe to — iad latte that that 
transaction pertains to. ; 

The interactions among the transicons wget thatthe proceing tobe performed 
exhibits a high degree of locality of reference. H the lems are grouped s0 that all of the 
items pertaining to a single location are maintained by 2 single manager, the only 
transactions that require the participation of more than one manager are the S transactions. 
These transactions represent 05 of the torst vohume OF traneuetone tabs rin (though they 
probably represent a higher proportion ofthe proce ecamex they are mare complicated 
than the more: frequent transactions), | 
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6.iaP Transactions 


_ @1eOTransactions —— 6.108 Tyansactions 


A Joint Transnstion Graph of The leventeny Tranamections | 


6.2.2 Organising the Dats 


In this section, I consider several difterent-wayi Inwhich the data items could be 
assigned to data managers. Each of these organizatiits) fof te diti'bhse is discussed t0 
show how. the four types of transactions would be‘fierformed in such an-crgantization. The 
actual choice of implementation would be based on the desired level of availability for the 
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data items as well as the cost of performing the transactions and the processing and storage 
‘Capacities of Ame sites deplaing the data er processes... 


A simple organization for this data base would be to > assign all of the items 
pertaining to one site to one data manager proces which executes at that ste. A joint 
activity graph of the four transactions as performed in such 8 an organization is depicted in 
figure 6.3. The graph shows:that inthis organization, the only type ofitransactions:requiring 
| ‘communication between data managers are the & transactions. Ait of: the other transactions 
ean be performed by:one of the managers-alone because aitof theitems involved in any of 
the-other transactions. are.under control.of a single data.manager.:° | 


| Figure 6.3 : 
An 1 Activity Graph for a Simple Pate Bese Ongentention 3 


Assignment of Tiewss'te Maxacers: 


a es a | ; “ Ma . 
QOH, . QOH, QOH 
QOO QO}, QO02 
RQTy RQT, RQTo 
DQOH, DQOH, DQOH, 


QS Qs; - -QUSe 
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In this orgawization, most . transactions. would /-eequire no interrmanager 
synchronization at alt, and the rare: §. transactions wolakdtequire: todling “with any 
synchronization network of the date manager processes (because of the cycie involved in each 
S transaction). Because none of the ier in this organization are replictmd, this 


es By £ 


erganizaton require the minimm posible storage space. 
While: the. tramanctioms are infrequent, the mecessity of bechiny; wiaite performing, xn 
" S teaneaction ie undesirable. Locking makes the 2we-siees involved: tn-pertoening: 41'S 
can be used to reduce this walmerability so:an acteptable jewel seh: as:esing:potywaties {as 
will be discussed in a inter section), or running the S transactions at a time when there is 
little other activity, such as after the stores have closed. We can avoid the necessity of 
locking fer the & transactions by reorganizing the data base. 


The arcs from Mt; to Mg and ee result from the necessity to update the 
avoid this. dependency, by maving the item JS, from manager Mf; to manager Mier A 
joint activity graph for the gatulting organiz ae i own in Figure 6:4 


In this orgamtzation of the data, agath all transactions except for the S transactions 
are again completely local ta one of.the data managers. ee rar are performed by 
two of the managers and require communication. Unite the previous organization, 


however, this communications one-way, such that if hierarchy of managers is chosen such 


that Mg aise 2 dmemaane of Mien the & womens am be prt wine 
locking. 
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Migare 6.4 


. An Activity Graph for a more efficient # Organization of the 
ata — i : is 


Assignment of Items to Macagers: | 


Mo: My Moa 

QOH) QOH, QOH, 
Q00, QOD} 
RQT) _RQT RQT5 
DQOHp ‘DQOH, DQOH2 


Figure 6.4 shows the joint activity graph for 3 locations in the hierarchy of | 
distributers and supermarkets. In a reat application, there would be several supermarkets for 
each local distributer, and several local distributers.. This makes the joint activity graph 
a more CODON EY: as shown by Figure 6.5. | | 


Sigare 65 shows the joint activity graph for this organization of the data, for a 
system in which there are four supermarkets (Thus four Mo managers) being supplied by 
two local distributers. Each manager and each transaction is, Iqbeled with two subscripts, the 
first indicating the level in the distribution hierarchy and the second indicating the location 
at that level to which the manager or transaction pertain, The graph is hierarchical, with 
an arc from each manager to its parent. Notice, however, Shing using the apparent hierarchy 
in Figure 65 as the synchronization network would not allow the transactions to be 
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performed without locking. Each manager (Exogpt for the Mg managers) must obtain 
information from its chikises, in the bierarchy to perforan the § trpnsactions, Ifthe arcs in 


Figure 6.5 were reverse, then the transactions cool! be performed without locking by ering 7 


the hierarchy in the joint activity graph as a synchronization network. I will refer to an 
Activity graph of the form of Figure 65 as an inverted hiezsrehy. to-distinguish it from a 
hierarchical graph im which there ian arc running froin eich manager to each of its 
| children. 


s Figure 6.5 | 
_& More Complete Activity Graph . 
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Given this organization of the data base, we must: chose a synchrenization network 
‘ that allows the transactions ta be performed with: Ten piencene’ Cheapest 4. While the four 
| Classes of transactions described here do not involve any transactions that:access a targe 
number of items, presumably in a real inventory control system there: would. be other 
transactions much less frequent than those in me four classes ween perio functions such 
as changing the petamercts DQOH and ROT, | or allowing a user ‘to obtain a snapshot of 
the quantities of some item in. the various locations In order to ) provide ae ability to 
synchronize ” possible transaction on the data, the i organization of data managers must be 


hierarchical. 


Any hierarchy of data managers that is consistent with the inverted. hierarchy 
defined by the arcs in the aie activity graph must be some linear areas of the nodes. 
_ The conditions that M 2,0 be a descendant of all managers, and that some Process be an 
ancestor of all managers, and that there can be only one path between any pair of managers 
force a: linear ordering. This is not a very desirable organization for synchronizing ¢ the 
transactions, ‘because the message sent from some © manager My to Mgt 1 in performing an S 


iyo esa? 


transaction may have to be routed through many other managers that do not otherwise 
‘ participate in that transaction. This makes bs transaction ‘expensive and vulnerable to 
failures, however a failure occuring during s an 18 transaction does 1 not ‘unnecessarily delay ; 


other transactions, because there is no locking. 


_ Another alternative is to abandon the ability to perform any arbitrary transaction 
: and restrict the synchronizatien mechanism to.acting on the.four.classes described above. If 
we are only interested -in. perferming these four, srapenctions, then the only conpusnication 
among managers that is.needed is that described. bp. the joint acvity.geaph of the four 


transaction classes. A non-hieratchical synchronization network could be used to coordinate 


es - 
for a syndbrontztion wework soutined in Ohapur 3, :th tas-beily ene path Detween any Pwo 
processes), and ‘prowbdes <i sf che wenesanry wonnmenibaien pattie 20 curry ‘ut transactions 


. -fhrom tre four chu. 


"Than the dam managers could be ficly organized im an inverted hierarchy. All 
of the transactions except the S transetions woukl, as before invelve only one tata manager. 
“To perform an S tranmction, a menage woul! be st the manager M, holding the fem 
Qo0;.. This manager would tain the value of this tem and send it to the manager M,.7 
which holds QOH,,, and QUS;. a ais aaa eae cca 
ee ee 


- "Thin ergantzation ofthe managers ls clearly mest efficient in term of the arpount of 
_ locking and the number of manager sent, but kas merficed the ablity to coordinate other 
"kinds of transactions One woul! protably not want t0 chotee an organization of data 
“managers in which ft i mot possible to synchronise any arbitrary transaction, as this may 
make it very difficuk to implement new kinds of transactions. Stil, in a situation in which 
the transactions’ and the dam base are permanently fined, such a1 a distributed rocket 
guidance system, choosing the topical organization of the data managers without considering 
{he kinds of new transcins at coul be desrd i ot «problem. 


6.2.8 Replicated Organizations of the Data Base 


The ‘two onganisations of the thuen ‘base described store have 2 sifigte copy of each 
data item. in. this section, 1 consider organizations im which. some Of the’ data items are 
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transaction will be delayed due to a site being inaccesdtble); or in order to etiminate locking 


Be nts 35.23 


.. by making more of the transactions have indepeiiderit ‘cori 


One could, by replication, make all of the transactions have independent components. 
This effect could be achieved by making sure-that whenever a maithger holds a copy of 
Some item I, it also hekis copies of alt of the tems ‘needed: by the trantactions thitt update I 
in order to make that update. For each:item § held bya manager, -that'manager rnast also 
hold copies of alt items from which arcs:in the joint: transaction ‘graph point at-E: Thus, in 
effect, each manager hokding a copy of I must hold copies of all: teewts that are linked to I by 


7 a chain of arcs: in the joint transaction graph: The: joint thangattion «  aph of Figure 6.2, 


indicate that a site holding a copy of the items QOH; or Q00, must'abo hold copies of the 
items QOH; Q00;, RQT} DQOHy af and WS) | for am Md s , pease (Of the chain of arcs 
linking these items to QOH, and Q00,. This present an awkward ‘problem, as it means 
‘that in in order to make all ‘transactions have Independent components a Single site must hold 
‘copies of all of the items and therefore must participate in all of the transactions. It 
_ therefore does not seem practical to avoid locking through this approach. 


__ This particular application appears to have'tittlé need for replication to increase the . 
| “availability of data items. The transactions that-are‘most critical'to perfor quitkly are the 
- P transactions and.the-O:tranaactions. White we could répliéace the'QOHo items iti order to 
increase the availability of these items, there seems to be little point in doing 50. Because the 
P transactions are by far the most frequent, replicating the QOH, oe would add greatly to 
the amount of communication and :possibly the-amount of computation: required. A more 
appropriate approach might:be.to make the sites which hokd: the QOHg items highly 
reliable. Another approach ‘that could be used:és to use several sites to hoki the data items 
pertaining to ack supermarket, partitioning the items so that for each product, there is one 
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site that holds ail of the dems that, pertain. te. that, product. \ This approach may. allow the 
individual sites to be senaline, simpler, and.mere snfiable then: @ single ete managing: all 
items for a pasos 


It would ale, premsabiy, be impoctant that O tranaatons be enecuted prompt, to 
insure that supplies of preducts remain. adequate. In. onder ¢n_make:the O transactions tess 
vulnerable. 20. failures, we -coukl.-replicate:-the: items aceeaed: by: the © transactions. 
Unfortunately, the ©. transactions: at-each. location acsesn manyrot the:itertts for that’ location, 
including the QOH .kems.. Tus, replicating, denis: te,eonke Ob teanenctions: more retiable 
‘would enake many of she drnreneane moseenpansian, dem to: tee anemia npate the 


"Another organization of the data that might be ened Isto replicate the JS; and 
Q00; items so that M, and M,, each have copies of both veers, Figure 68 shows a feat 
“activity graph for this organization. In this organization, My and Met each have copies of 
the items Pertuining to orders sent from location it cation to. “This organization does not 
provide any reliability advantage over the first organization considered in ‘performing the 
four transactions, Haying the QJS.aad QOO items seplicaed: may, however, allow the 
"human managers in. charge. of shigping and. receiving. at.the sites.te sletermine.the status of 
orders more easily, even if a failure interrupts coramuniention Petvepen locations. 


6.2.4 The Use of Polyvaues 
Another way of increasing the availability of the -datacitems in the event of a fatlure 


is to use the polyvaive..mechaniam described: :ier Ghapeer :& Ak noted: above; the “P 
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; Figure 6.6 
An Activity Graph for a Redundant Data Base Organization 


Assignment of Iteins to Managers: 


Mo: My Mog 
QOH) QOH; QOH» 
QO00 9 QOOo, ; QOD. 
RQT 9 -RQT). RQT2 
DQOH) DQOH, DQOH) 
QISp US; QS. 
QISp QIs; 
QO00o QO00o 


- base is rare, it is possible that a failure during one of the S transactions could delay access to 


the items used by those transactions. This coukl in tun delay wther transactions. 


‘By using the polyvalue mechanism described in Chapter 5, wean avoid this delay. 
Two factors suggest that the polyvalue mechanism would be effective in eliminating 
unnecessary delay of transactions local to one site by failures of ether sites... First, many of the 
transactions Geen only loosely on the actual data vcr values. The o transactions, for 
example, make a decision of whether or not to order that depends only loosely on the items 
‘read. Second, very few of the transactions eee locking, thus the probability ar a 
transaction requiring locking will be interrupted by a failure se small 


at a 


168 - 

Notice also that most of the transastions teeuld not propagate potyvalues that have 
been: introduced. inte: the data base The P transactions and: R:teageactions de: pat propagate 
information among items in the data base, while the other two types of transactions may 
propagate a pofyvalue to at most one new item. This means that if a polyvalue is 
introduced, it wiff not cause uncertainty to be roves rouge tive data base. 


Whether or not the potyvalue mechanism should be used for this application 
depends on the actual cost of implementing polyvalues. ‘in tetris of the extra checking that 
| must be performed in the course of performing a transaction to handle the possibility of 
polyvalue inputs), and the concern for reliable operation, 7 The cost of implementing 
polyvalues is not likely to be high, but the benefits are likely tobe small, as so little locking is 
performed in this implementation 2s to make it untikely thet a polyvalue will ever be 
produced. | | 


6.3 Comparison with Other Mechanisms. 


Several other mechanisms could. be used for performing synchronisation of the 
transactions in this example. This section briefly sonperss some of the on mechanisms 
‘that have appeared in the literature with the solution described above. 


63.1 Comparison with SDD-i 


As this example is derived from one used for the sDD4 system for synchronization 
of distributed data bases, it seems natural to begin any comparizon with SDD-I. This 
discussion presumes that the reader is basically familiar with the SDD mechanism and the 
solution to this probiem using SDD-t. 
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Using the analysis and protocols of SDD-1, one concludes that the P transactions and 
the O transactions can be performed by the simplex (pl) peetneol.--This protocol requires. no 
Jocking and in fact closely resambles.the protocol used. to. perform: these transactions: in the 
solution described above. The other two transaciion.classes, however, nequire the p3:protocol 


a. of SDD-1. This protocol performs locking, by: forcing. the-variqus.data.managers to perform 


, fansactions.in time-stamp order.. Thus SDD-1 locks far: two of, she. four transaction. classes, 
while my. mechanism Jocks for only one... The, :easqn,,that.twe..of she transaction. classes 
| ‘require locking in SDD-1 is becayse the analysis technjanes sept by SDD-t-do wm. recognize 
that the R_ transactions actually have three independent enpenems: While these 
_cornponents must tbe performed atomically with respect to other transactions, there is no flow 
of information among the three components, thus they can occur in any: order with respect to 
each other. The more fine-grained analysis used in the ‘mechanism of this thesis discovers 


this fact, which allows these transactions to be performed ‘without locking. 


The locking protocol used by SDD-{ is similar in cost to that used in this thesis, if 

the SDD-1 mechanism is implemented simply and without. regard to. failures, ,.Both involve 
_sending a message to each pf the data managers. which. will:be involved in the frensaction, 
_-Fequesting that.the data, manager set aside some time to, perform. the transaction, and. then 


performing the transaction with as many additional, messaga:¢s are.neaded tp move the 


data. - 


The robust. implementation of the SDD-1. protocols [Hagamer78), however, is very 
| complicated, and may,.involve many extra._messages Ak appears, be quite difficult.to. be 
sure that the transactions, are. performed in timestamp. roma olerings Sak failure. 
to stop.all transaction Proves... get eg Sas 


~ [7 - 

The robust implementation of. the: SBD9:promedts stempts to minimize the 
probability that & falliire wilt make data wuccestbte divengl’the use of aborwuble locking, 
and a voting strategy to duermine when & transaction’ pasedey its conimit point. Using these 
techniques may greatly increase the nurhber of messages that trust Be sertt afd the processing 
“needed (© implement transections wsing the locking ‘pronottth’ “liv contrast, using the 
polyvahee' mechantum to-tncrease the avaitebitt) of dats that tiast be 'focked does not greatly 
“increase the cost of performing trartsactions if no'fsharey occur. bec doublcedenaies 
ee cee en Sern ee eee 


| In surnary, the protocol wed in thi thei are ety toe slighty les cn (i 
terms of processing power, and messages sent) to achieve & om arable level of robustness 

" than the protocols of SDD-L This doesnot mean that my mechaniam is ways tes con 
_ than SDD, as the cost of both mechanisms depends strongty on the application. | 


6.3.2 Comparison with Gray's locking strategies, 


Another distributed ‘concurrency controf tirechanisin' i described in a set of notes by 
Gray [Gray77). These notes desctibes a° mechaiiism inv wilich a set‘of sites, eaich of which is 
‘eapable of “synchroftizing © local transactions, ean - todiiieainttied “to " perfotm muki-site 


items in the data base to various sites. 


| The protocols used by Gray require locking whenever two’ ot more sites are involved 
in one transaction. ‘Thus’ the 8 and 1 trantéctions: wotAd’ require locking in this exampte. 
The locking mechanisms proposed by Gray are a metHanisrs for recurtling ‘the locks held by 
each transaction at each site, and a deadlock detection mechifiisht to éxcupe from ‘settings of 


-i1- ; 
focks in which no transaction can proceed. The-cost of setting ‘the locks needed to perform a 
transaction is similar to the-locking mechariishs ‘of SDD-Fand this thestl: |” ; 


The problem of deadtock -detection, Howevef; adds to the tost of Gray's scheme. 
- Deadlock detection requires analysis of the sets of locks’ fiek! by’ alf transactions at all’ Sites, 
and. may be quite costly in a’ large system. "Grey agate “hae ede foek detection: cai’ be 
partitioned, so that deadlocks among small gretips of ‘sites cant be detected tore rapidly and 
with less computation than deadlocks tabboldas 5 a \ large number of sites. This strategy is 
| likely to work reasonably well in this application, as cach transaction involves only a small 
| number of sites. _ Deadlock detection sil represents an. additional cos cost in operating. the 
system, over that of using the protocols of spp and this thesis which use presnalyes of 


‘the transactions to avoid deadlock Sitdiationk: 


6.4 Summary 


_. This application (the distributed supermarket inventory system) is typical of the 
kinds of distributed information systems which. this thesis addresses. The analysis shows 
that the transactions exhibit a strong degree of locality of reference, and that most of the 
"transactions can be implemented without locking. The choice of which of the data base 


organizations and synchronization hierarchies to use for this application depends on the 


7 “concern for reliability, and the desire to maintain flexibility to perform transactions other 


-than those initially planned. The overhead of synchronization in the organization in which 
the hierarchy parallels the hierarchical organization of the locations is very small, as very 
few transactions require locking, and no extra messages are used for the synchronization of 


‘transactions which do not require locking. 


| - 172 - | 
"The potyvalue mechanisen described:in chapter five can be used in this application 
to minimize the probabslity that a failure will delay transactions... 


| The implementation of. this application. using the techniques of . this thesis was 
compared with, two ether distributed data base concurrency control mechanisms This 
comparison pointed out that the techeuques ef this thesia were libely.tn be lass custhy than: the 
| other mechanlums in achioving a compara lve cebueren. 


| ‘in concluson, the syhrontztion machanam ofthis thes can be sen to be efficient 
and robust for this application. ‘At the same time, the mechantim allows a great deal of 
_Mexibiey, lating the sytem designer trade off the dee for retabiity and eficiency agains 
the ability t® incorporate unplanned transactions eal. i 
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Chapter?” 


‘Conclusions and Areas for Further Research 


T his thesis has presented a mode} of synchranization.of transactions in q-distributed — 
information, system, and several mechanisms for. prowiding,;such synchronization. . This 
chapter summarizes the important contributions of the. thesis tp) this field, and suggests some 


areas for further investigation. 


7A Summary of Thesis Work 


The work of this thesis has concentrated in two areas: development of a | model of 
computation in a distributed information system, and development of specific mechanisms for 
concurrency control in such a system. The major ideas ot the e thesis in ‘each of these 1 areas 


are : summarized below. 


TALL A Model fee Distributed Computing. 


The process model of distributed computing presented in Chapter 2 isa framework 
in which computation .in a. distributed information: spmem can: be discussed, This model 
“specifies that the effects of site failures or communication: fadkures are: lost or delayed 
messages. The thesis discusses techniques that could be used>te: provide an: implementation 
of the concepts in the process model for which the effect:of failures are confined: to these 
specifications. 


-|4- 
1 developed tuo hasie satis for_syachronizing trancactions described in the 


Process modet: locking and sequencing. Sequencing achieves the goal of pertial operability 
defined in Chapter 1, white locking may allow a failure of one site to delay a transaction that 


is local to some other site. in Chapter 4, I demonstrated that locking was needed to correctly 
pater ais foal aera Casein 5 proseielian epponet  e es ee 


by a failure at some other site. Taken together, these rent se as 
to achieve the gost of partial operability while correctly ijnctiie 
the possible effects of faihires as specified in the process model. 


TAQ A a Concurrency Contest Mechanism 


Chapters three and four ofthis thesis presenta hlerarchical mechanism S9 coordinate 
transactions. This mechanism has several interesting properties, First, it is quite simple to 
“describe, and relatively simple to prove correct. Many of the synchronization mechanisms 
ddexcribed in the lierature are quite complex, and correciness profs for these mechanisms are 
very long and. complicated. The simple implementations of the protocols described in 
Chapeers 3 aad ¢ sugges that sy enechaninm st toiiey anit tga tm am 
actual distributed information system. 


A aero preperty of my scheme ts that-it performs well when the patterns 
“of accesses to items in the distributed ‘data base ‘show a: strong ‘Jecatity of refererite. The — 
“mechanism can be tailored so that frequent -ransetions’ eaquire’ fittle overhead ~ for 
synchronization. The: mechanism can also be desigitd= so 45 to: avoid locking whenever 
possible. The thesis describes analysis techniques that can be used to assess the cost of 
performing the most frequent or important transactions. This analysis can be used to choose 
an organization of the data and the synchronization network so that these transactions are 
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performed efficiently and reliably. The mechanism provides.coggect synchronization for all 
peneecinhs even those not anticipated in the design, however unanicipared transactions 


may be much more costly to ial and more oo to be delayed by failures. : 


Chane 5 of the thesis presents a novel solution. to. the, problem .of unavoidable 
| dey caused by failures during the execution,of a transaction. using locking. The polyvalue 
ee by that transaction can not be ectaes: fanny to.a failure. With this 
; mechanism, important transactions that must be performed promptly are, in many casas, not 
détayed by the locks set by other transactions. The protocols presented for manipulating 
polyvalues again are most efficient if most of transactions are local to one or to a small 
number of sites. This pears of locality -of ‘reference. Spears to be true of meny 


applications. 


| The model and mechanisms of this thesis. shed. some light on. what is a very. poorly — 
understood area of computer science. They do not by any means provide a complete solution 
to. the problem, and in fact suggest. several interesting research problems. 


7.2 Areas for Further Research 


There are a number of ways in which the work of this thesis could be extended to 
provide a better understanding of synchronization in distributed systems. These include the 
investigation of the applicabitity of the iesied model to real physical systems, further 
investigation of applications Deter techniques for ceamatructing me _synchrontzaton 


+ 


hierarchy, and implementation of the protocols. 


- [76 - 
7.2.1 The Applicability of the Process Mode! 


The results of this thesis are based on the semantics of failures in the process model. 
In particular, many of the resus are based on the notion that there is no single event 
detectable by two processés simuttaneously. While I strongly believe that this is true of any 
physical system, there may in fact exist ways of impleténting comiinieation_tn which the 
sender of a message can know for sure whether ot not that massage was tebdived. If this is 
the case, one might be able to implement abortabW Kicking as described in’ Chapter 5,, | 
RSE nee eee ee eee ee in the 


 Hterature: 


_ Another retated area for investigation is that of ways of including the effects of 
failures in a model of computation. In the process model, I assumed that a failure could 
delay any message indefinitely. It is possible that some less pessimistic assumption about 
failures would lead’ to'a workable model for'a distributed inforniatior system.” One might, 
“for example, assume that no more thar’ No sites fat Concurrently. While it would be 

impossible to implement a system so as to conforni to this aseiinptton, if the probability that 
“the assumption is violated is sufficiently small, then a distributed information system based 
on the assumption may be acceptibty retiable; and may:¥e simpler to implement. 


7.22 Applications 


This thesis makes extensive use of the aorta senile that spplations 0 of a distributed 
information sytem will exhibit tocality of reference in their use of data. This seamen 
appears to be true of some planned applications, however more careful ‘statistics of actual 
applications may be needed to confirm the validity ‘of this assumption, We may in fact 


epee 
discover that the flexibility of a distributed information synem’ ‘wilt encourage different 
Srgercnions of information that do not exhibdit the ee 


7.2.3 Analysis of Transactions 


The thesis presented’ techniques for determining the cost of petforming a transaction 
(in terms of the number of mone a = sacs erate iarisie 
given a destription of the most frequent transactfons to’ be peiformed, were given. These 
| guidelines are not, however, detailed algorithms | that design the 3 ynchronization, mechanism. 
Considerable effort and ingenuity may be needed in choosing an optimal, or near optimal 
synchronization network, and in choosing | the assignment: of ata, items to data manager 
/Sprocestes: These problems are similar to many others that occur in managing resources in a 
| ths for designing. a. distributed 


aa pe system, _—_ it would seem likely that good algorit! 
information datas using the hierarchical synchronization | mechanism of this thesis could be 


derived. 
7.2.4 Implementation of the Protocols 


Finally, the thesis presented only a few simple implementations of the synchronization 
Protocols. Improvements on these implementations can no doubt be made. One area that 
seems of particular interest is using the hierarchical synchronization protocols in a computer 
tailored to managing data. The hierarchical ‘metronization: mechanism presented here fits 
‘well. with’ the proposed. madels of memory fora data base niachitie: This thechanism may 
lead toa very efficient implementation of sucha machine. 
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| Another implementation issue that benrs. further investigation is. the design of 2 
communication network that. supports atomic, broadcasting. In Chapter 2, techniques for 
Using = broadcast network te iaploment the memage forwarder protocel. were presented, 
The need for a coordinator ste is a weak point in this scheme, as fale of the coordinator 
‘ite stops all atomic broadcasting. _1t Ss possible that, the fuuncaiqn, ef the coordinator could be 
implemented in each site's network interface, in such a way.that a single site failure would 
broadcasting, itis Hkely that broadcasting cou be made efficent and. highly reliable. 


Several mechaniams have bem developed to rain several identical copies of & 
redundant database. As noted in Chapter 4, this problem appears to-be somewhat simpler 


es than that of synchronizing transactions in a distributed information system, because each site 


has a copy of every item. The tachniques that have been developed to manage replicated 
‘data could be appiied to maintaining copies of the process sate and mesiage queues of a 
process, in order to obtain a more robust implementation of a process “Application of . 
techniques for maintaining duplicate data bases to the implementation of processes would be 
an interesting research problem, and would lead to a move robust implementation; of :the 
concurrency control mechaniam presented. im thic thesis 


This chapter. has presented a.summary of the resatts of: this thesis, and preserted 
some of the open questions that this thesis, leaves. unansweretl... Many of the conciuslons of 
this work are not decisive, however | hope that.nry: work: Se ee a 
on a very murky and poorly understood field. 
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Proofs of the Protoools 


This appendix gives a more formal deftriition of some of the concepts in the body of 
this thesis and proofs of sone of the results.. For sirnpticity of description, the definitions and 
proofs. in this appendix are for the version of the micstage-fevafder peotiicol in Wiileh’éach 
peo ieee eee ee sais Is a common ancestor of all of the 
° Of the protocol described in 
Chapter 3, each message may travel up the hierarchy in several hops. This difference does 
not effect thes eeouke preven these, se long: 4s only: the-smmenge: fetaptions: Which: take place in 
the, distribution of a broadcast. after that broadcast. es sanshe the common ancestor are 

used in determining the < ordering. This condition is consistent with the use made of the 


eps that take place in the sisirdbenion ‘ofa: meciage: after) NaN reached the common 
ancestor are the only ones which have effects that couki be observed by process steps related 
to other messages. ) ee | ee es - 


Al Formatization of Atomic Dvedcatng 


Definition: ‘For each process ‘ hoc asi uteta: scalp uh 
“% ; pr. 
PREY SEARS Ns CoP ta CIEE Sp em ec 


Definition: A broadcast B = {[b,, sot es eee reer ye emer 
process p, asapartofB} = 


Definition: For each message m, let B(m) be the broadcast message from which m 
was derived. The set {m[B(m) = B} for some pean broadcast B 


Mo was received at p.. 


Definition: 


Def inition: 
. Definition: 
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then contains both the component.messages of B and any other 
messages that are received in distributing those components. 


For broadcast messages B, and Bo, there>is' an-ordering <, which is 
defined as B, < Bo if ap, by bg such that Bib) = By and Bib) = Bo 


Breieaniog is atomic iff < is cyl free. 


Let ~ be the Synchronization networ k. relationship, which is a 
relationship. among pairs of processes andi musi:satisfy the following 
constraint. The Gah defined by ~ must have no directed or 


_Undirected cycles, Thus,.thare dogs not.extst-2: setref three or more 


processes py, ...fp, all distinct, such that ae a eae Peet | or Pew 1° ~ by 
for all icn, and p; and p,‘are-tetatéd ‘by :: 


“With these definitions, we can now define the message forwarder protocol by defining the 


process 9 ‘specifications of the forwarders. 


Definition: 


Definition: 


The process step amare ofa gems forwarder is nerwetibr 
a function F(B): aks 


F(B) = {((X(B,p),p] | iy P A The set. (py) is non empty} 


-where B is the message, received. in a. process, sep,-F(B) is the set of | 


pairs, each of which lists one of the t messages produced by that 
step and its destination. pracess,.and:, ) igrm set: describing . the 
contents of one of the output — = the eae wee which is 
constructed as follows: . . 


X(B.p) = {Ib.9) | bg) € BA p ~* g} 


Communication between message forwarders obeys the constraint of 
Sequencing, which can be stated as follows. If b; <¢ bo for messages 


by and by and message forwarder f, and if (h’.p] € F(bp, and [b’o, p) 
€ F(bo), where F is the protocbl ee wach then >| 7 "Bo 
after both b', and. b'g have. bees: saa ai dle 
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A.2 Proof of — es 


acacia cause facaatea cease disease We Sacenctets dG 
message atomically, we must show that the < ordering in any system state reachable by an 
execution of steps specified by the protocol is cycle free. To do sa, I will show that for any 
secp permitted by the protocol ifthe < ordering i cy fee before that step, then it will be 
so after also. Because the starting sate (before-aniy mesunges ‘have been received) has an 
empty < ordering, whicte te seria cycle: fret, ey induction ter ey state reached by following 
the protocol the < ordering willbe cycle free 


"Before proceeding with the prof, 1 would like male an observation about the 
protocol that will be useful in the proof If two processes. p and ¢ have each received a — 
message derived from the same broadcast B, then there is a path in the graph defined by 

the~ relationship connecting p and ¢. Each process on that path has also received a 
message derived from B. ‘This is true because the graph has ino cyeies, thus there is only one 
path between p and ¢, and the protecol allows each broadtast #0/enter the network at one 
point, from which it must reach all recipients It ts not fiessible for two processes to have 
seen messages from a broadcast B untess all of the proceises err the path between those two 
. processes have also seen messages from B. | 


In each step of the Protocol, some message m is received at some process #, possibly 
adding ordering relationships of the form’ b(m’)< &(m) for messages m’ previously received 
at p. We must show that introducing the relationship b(m’) < b(m) for any message m’ 
_ previously received at p can not introduce a cycle. The proof will be divided into 3 cases, 
depending on the origin of m and m’ i 


CASE 1: m was not sent from a process P such that P ~ ». In this case, m is the initial 


si: 8H - 
entrance of broadcast message b(m) into the network of message forwarders. Therefore, 
before the reception of m, there were no ordering felatiinships in'< involving b(m), so that 
the reception of m could not introduce a cycle in-<. 


CASE 2: m and m’ were both sent by some process P such that P ~ p. In this case, the 
“process P must have received a message-M such'that ‘B(M) = Bim); and ‘a’message M’ such 
that B(M’)“= Bim’) in the process ‘steps which produce ‘mn ‘and 'm’. Because of the 
“ sequencing of messages between P and , the messages fiv-and: rif must ‘have been sent by P 
_ in the same-ofder that they-were received at p.  Fhus the ordering retationship b(m’) < bm) | 
held before the reception of m (because of the réteptions-of Miarid Mat P) and’ therefore, 
by the ‘assumption that ‘ne cycte existed before tive reception of mi, oS ‘is created. 


$B 
rte 


CASE 3: _™ was sent by some process P for which P- ‘ad 7) but ma was not sent j P. This is 


“the most aifficutt case. To show that. no cycle is introduced a2 the reception of m in this 


. case, I will assume that such a cycle is created and show that this assumption leads to a 
contradiction or a violation of the conditions of the protocol. 


‘Assume that the reception of m creates a cycle in the < ordering. : Then prior to the 
_ reception of m, it must be the case that there isa sequence of broadcast ‘Messages <By, -. 
By> such that By < B,,) for] <i<n,and By) = B(m), and B,, = B(m’). Consider now the 
set of processes Pp -» Py-z at which these broadcasts were ordered. Now by the observation 
noted above, there exists a path in the network from cack of these processes to the next 
process in the chain. Also, there exists a path: Between pyiand 'P, as-both: have received 
‘messages derived from b(m), and there exists a path betwee p,., and p, :#s both fave 
‘received messages. detived from: b(m’). Thus:becatse of: this:-cteate of broadcasts, there must 
exist a path between P and p. If the. path. implied toystive ‘chain Of broadcasts does not go 
through the direct link between P and », then we have discovered a cycle in the 
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synchronization network, vislating the conditions of she:protecel, Jill show thet af that 

path does.go through the-dinnet dink detwaen the two peaceses, then either the sequencing -of 
‘messages between P and » ‘thas ‘been violated, ora cycnoninted dn the < selationship afore 
the ee oft at p. | | 


If the path between P and p isdises a lea eee broadcasts includes . she 
_ direct link, then some -begadcatt, call it By taust, hauedoeen. seen by beth .P, and p, and 
furthermore, we knew that? caus hawe cogehyed @ anessage aderiued Score Band ase. result 
"tent a rmessage top. The terondansts 3; and: Bian) smust shape dboen andered by eanssnge 
_ ecaptions at -P, and Bim) < By, as-atherwise: these.arould: -be oa apse in the .chaie. of 
broadcasts... Now by sequencing, the reception .of m at. neue precede ‘any -veaspsion of the 
message derived et eee as we know that a message derived from Bj 
must have been received at #. This contradiction demonstrates that iti impossible for a 
cycle to arise from the reception. of m at ifthe only path between and p bth dives tink. 

" Thus another distinct path must exist in the semmenaeed network betweer 


P and p, 
forming a cycle with the direct link. | 


This completes the proof-of the third and sc thasthesycrontaion potoo 
of udealesiaten: heat aeanersincaneces acid 


P ah “3 tbe ign Pas the 
A.8:Correct ‘Relative Sequencing of Broadcasts = 


Jn this ‘section, .| «demonstrate shat the protacol destribed: in “Chapter 3-far atomic 
message: before :recei ving:some menage that.en "shauhi:fellow’,. ‘ihet:penot sill be forthe 
simplified case.in-whith she-~ salationshipsis a:simppletiierarchy. © 
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Recall that the "should follow" relationship among messages was defined as: Each 
message m sent by a process in a process steps: shauld fellew a: message m’ whenever: 
a) There is a message m” received by p in process step s or in a step 


that preceded 5, and m’ and m”: areceomponents:of the same ° 
broadcast. 


OR 


b) There is a message m” received by pin = sor ina aed that 
Preceded s, and m” should. follow.m’.. 
A ‘key factor in this definition is that if m should follow m’, then some process must nave 
received a message derived from b(m’) Using the message forwarder protocol of Chapter 3, 
if any process has received a message derived from a broadcast message B, then for any 
_ process p. if p will eventually receive a message derived. from: B, then that message must be 
represented in some message awaiting reception at por. at one. of the ancestors-of p. This is 
true because each broadcast. message. enters: the. hierarchy sonce,,aad. all components flow 
_ downward in the. hierarchy. fram the point of entry. No-component an .be received before 
"the message is entered in the hierarchy, and. once a sessage:is- entered, each component is 
either above or at its ultimate destination. 


T will now prave.the claim that for any. message m, there can-be no message m’ such 
that m. should follow m’, and the ressage conbeining.the-camponent containing: m' is above 
that for m in the hierarchy. This, combined with the observation about the .message 
forwarder protocol described in the previous paragraph, is sufficient br prove that when a 
message m is received at a process hn no menage f m’ that should foliow m will subsequently 
‘be received at p. 
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' The proof of this. claim wilt be by induction.  Initigity, the claim is true, as there are 
no messages. We must show that in any state for which the clei is true, the reception of a 
"Message m at a process » as specified by the protocol cannot cause the claim to become false. 
7 There are two cases: one where m was sent by the parent of f and: one where m was sent by 


' some other process. 


CASE I: m was sent by some process P such that P ~ ». When m was sent, all of the 
messages that m should follow must have been in the titrarchy (or already reveived) and not 
' above P in the hierarchy. Therefore, because of the sequencing of messages between. p and 
P, messages that m shoukd follow will nt be above p in the hierarchy when m is received at 
~*~ 

CASE 2: m was not sent by-the parent of p. In this cass we must consider the messages that 
m should follow. These are afl components of each. broedcast-teisage B for which s; the 
sender of m, had received # component prior to the sending of if: T'he clatm was trie when 
Mm was sent, so no message that should follaw any of these Brondiasts could: have been above 
$ at the time that m was sent. Therefore; because # mest be an ancestor of s, there are no 
messages that should follow m that are above p when m is recétwed ‘at p. 


This. completes the proof of the claim, and thus the proof that broadcasts are 
SN a ok tern een ee eee See ee 


network. 


While the proof of correct sequencing of messages according to the “should follow" 
retattonship is somewhat involved, the principal of operation of the protocol is simple. Each 
message pushes the messages that it should follow along paths in the hierarchy as it goes. 
The protocol works because there is only one path between any two processes in the 
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hierarchy, so that no message can sneak ahead of its place in the sequence of messages going 


to some destination process. 
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An Analysis of the Propogation of Pulgvalues. 


A major area of concern regarding the polyvalue scheme presented in Chapter 5 of 
this thesis is that failures may cause the number of items having potyvalues to become large. 
This would waste storage space and cause a great deal of extra computation by the 
polytransactions acting on the data base. This appendix presents a simple model of the 
dynamic behavior of a distributed information system using the potyvalue scheme. An 
. anatysis is given to show that with reasonable parameters for the expected transactions and 
failure rates, the number of polyvalues in the data base remains quite small. A simulation of . 
the system agrees well with these predicted results. 


B.1 A Model for the Creation and Deletion of Pelyvaiues 


‘At any point in the execution of a distributed information system, we can calculate 
the expected rates of creation and deletion of polyvalues, based en some assumptions about 
the expected transactions, and the failure characteristics of the system. These rates can be 
expressed as: | 


Creation Rate = Propagation Rate + New Failure Rate 
Deletion Rate = Recovery rate + Propagation Overwrite Rate 


Propagation rate is the rate at which polytransactions install polyvalues for their results in 
items which previously held simple values. New failure rate is the rate at which updates in 
progress are suspended, causing potyvalues to be installed. Recovery rate is the rate at 


191 - 

. which failures which caused pblyvakoes to be produced are recovered. “Finally, propagation 
overwrite rate is the (probably very low) rate at which. an item witha petyvalue is updated 
by a transaction producing a simple value. This ecours only if-a transaction eer an 
output. that is independent of the previous valus.of: en item. 


With some additional terminology, we can develop more precise expressions for the 
'. creation and deletion rates. I will use the following terminolegy -to despribe: the data base, 


the transactions, and the failure characteristics of the system: - 


_U - Update frequency (Updates/Second). This is the‘rate at which 
updates to the data base (not transactions) are made. U_ aan. be 
calculated from the overall transattidn fHite the | 
transactions which make pact and the average number of 
updates per, transaction, 


pee ce 


_ W- The probability of an update being delayed. by failuse. W-can 
be computed from the mean time between failures, the time window 
in which an update can be cag bs by a ora and me oly oe 
rate. dy ut 


I - The number of items in the ie heat 7 ; 

R - The recovery rate for failures. This is the rec fear of the 
- mean time ‘to recover failures (in seconds). ‘The description of 
. failure recovery in this. way assumes, that.the mean time tp. recover 

failures is =pererial distributed with mean ba! ie. 

be ‘Update ‘independence. This parameter is the probability that 

the new value of an updated item wilt-net? depend ‘én’ its: exact 

previous value. A value of 0 for Y indicates that the new value of 

an updated item always depends on its previous value. pe 


D - Dependence of updated items on other data iterns. This 


__ parameter specifies oa the average the nusnber of-date: deme in the - 
data base on wn aan Races meet 
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With these parameters, we can approximate the rates described above. In the 
_. expressions given below for the rates, P represents: the-number: of pelyvalees in the data 
base. This is a first order approximation in whicl the proportion of date items in the data 
base having polyvalues is assumed-0o be small; thus’ terms: volving ¢P/1)> have: been 
dropped. | 


Propagation rates UeDoP/T 
New Failure Rate=UeW 
Recovery Rate = P+ R 


Propagation Overwnie rate Us Pe VII - 


These tr canbe combined 2 give the epi of cng of the number o 
polyvalues in the duta base, yiekting: 


C3 = UsW + UsDsP/l - UsPo¥ I - PoR 


This is a simple linear differential equation for P which indicates that the number of 
polyvalues would follow an exponential decay from its initial value to the steady state value, 
given that. the parameters acctirately deicribe the Betavior Of the system. The steady state 
expected number of polyvales can be obtained by sting the rate of change equal to zero 
and solving for P. From this we obtain: 


_ UsWal 
P - TeR-UsY-Usb 


Several non-active propre ths soheton shoot be expt. Firs, it would 
seem that the denominator of this expression can go to zero, causing the steady state expected 
number of polyvalues to be infinite or negative. This situation arises when the propagation 
rate is equal to or greater than the rate at which polyvalues are removed through failure 
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recovery or overwritten. If this were the case, we would indeed expect the number of 
polyvalues in the data base to become large. | In fact, the number of. polyvalues in the data 
, base would grow so large that this simple first order analysis \ would no longer be correct, and 
the number of polyvalues would be limited by second order effects which 1 have ignored, (I 
have, for example, ignored the possibility that an item involved. ina failed update or the 


target of propagation already has a polyvalue, and thus does Not represent a new. polyvalue.) 


A second feature of the equation which may séem’ strange'is that it depends int a 
fioaictrivial way on I, the number of items in the data base. This is because the creation and 
deletion of polyvalues directly due to failures is not dependent on the data base size, while 
the propagation ‘terms depend on the ratio P/I. If I is very large compared to P, then the 
effect of propagation is small, as the chance that items with polyvalues will be used by 
transactions is small. If, however, the data base is small, then the chance that items with 
sol valiel will be involved in transactions is larger, and the propagation terms become more 


Significant. 


Another point to notice about these equations is that-they are stable, meaning that if 
the current number of polyvalues is larger than the expected number, the expected change in 
| the number of polyvalues is a decrease. This indicates that if some tasrOpre introduces a 
large number of polyvalues into the data base, the number: ‘should: soon decrease to the 
expected number, given that the valuns of W andR. are notte by the cena ties 


Table B.1 gives some typical values for P. ‘Several observations can be made about 
- this data. Decreasing R ‘causes an increase in the number of potyvatues, as would be 
expected. Anereneing W causes a ee increase in iad némber of polyvalues. | 
Decreasing I causes the number of polyvalues ter flee. The parmmeters ¥. and D have little 


effect, unless the values of the parameters are such that the ‘denominator of the equation for 


P is near zero. 


Notice that even for remonaby pesiiicfilre’ rte and recovery times, the 
number of polyvalues remains quite small These resuks indicate that the polyvalue scheme 
is feasible in a distributed Information system, and that the chances of a combinatorial 
explosion of the amouat of computation are very small The nest section of this appendix 
contains a brief descrigton ef 8 ston ofthe wor of pelyetue, which shows that the 


| : Table B.1 | 
Typical Predictions of the Number of Pelyvalues in a Database 
U w I R- Y D ee 
1 0.0001 -1,000000 =: 0.00 0 i. 0.10 
1 0.0001. 4990000. 6001 0 100 o.l 
10 0.0001. 1000000 «(0.001 0 1 1.01 
100 «=: 0.0l.«:19988,000 s(t 0 4 Nal 
10 0.0001 +=—«100,000.Ss«0.001 0 1 Lit 
10 = 0.0001:«« 00000 s«0.00 0 a 2.00 
10 © 0.0001 ~=—«#00,000Ss«é0.000 0 7 333 
1 0000 10000 0001 1 a 1.00 
10 0.0001 «20000 (0.001 0 i 2.00 . 
1 86.000 =i Sti«é 0 1 1.00 
10 0.001 1000000 0.001 0 1 10.10 
10 = 0.005 000000 «G08 0 4 80:50 
10 6.0001 1,000000 0.0001 0 d B00 
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B.2 Simulation of the Use of Potyvaues’ 


In order to verify that the approximations made in analyzing. the above model do 
not lead to : an inaccurate description of the behavior of the polyvatue : system, I constructed a 
simulation of the manipulation o polyvalues ina distributed information system which is 


based on the above model, but not t the approximations made in the analysis. 


. The simulation assigns unique identifiers to =n fallure ‘creating. a ‘Polyvalue, in 
order to distinguish them. For each item in the data base, the simutation maintains a vector 
containing the identifiers of the pending transactions on which ina: depends, refered to 
as the state of the item. An item has a polyvalue if its state is non-empty (i.e. if that item 


depends on a pending transaction). 


Updates are simulated at the rate U. Each such update selects a random integer d 


with mean D, and d random items from the data base. Some random item is selected as the 


target of the update. The state of the updated item is replaced with a merge of the states of 
_ the selected d items. With probability (I-Y), the previous state of the updated item is also 


merged into its new state. 


With a probability w, the update is chosen to fail. A failure is simulated by 
selecting a new identifier, adding-it to. the state.of the updated. temzand selecting a recovery 
time for the failure. Recovery times are exponentially distributed, with mean I/R. When the 


recovery time for a failure is reached, the identifier of that failure is removed from all item 


‘States. 


The limits of the siaanung program prevent the smuion of aed large-data bases, 


or very high update rates. However, for the parameters. rey can cal be simulated, the 


simulation agrees well with the predictions of the model. Table B2 contains the results of 
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the simulation for some semmple parameter settings. The nusbers of polyvaiues obtained 
"through the simulation were in general somewhat smaller than those predicted by the 
| analysis. This difference is primarily due to the fact that the rate st which polyvalues are 
created is smaller than that predicted by the modal, because the target of a flled update or 
se are at 5 petaamien way siear tare polyvaive | 


In conclusion, these results show that the polyvalue scheme is feasible for preventing 
delay due to locking, provided that resonable measures are taken to minimise the number 
of failures that introduce polyvahies 


Table B.8 
Rewalts of the Simulation of Polyvalues 


parameters S" ppedicted = aactual 
U w I R Y D Pp P 
2 0.01 10,000 0.01 0 i 2.04 2.00 
5 0.08 10,000 0.01 0 1 5.26 27 
10 00 10000 O08 6 1 Ht 95 
10 0.001 10,000 0.01 0 I Lit 0.74 
10 Oot 16,000 00 0 5 a) 19.8 
10 0.01 1,000 0.04 i 5 16.7 BS 


- 197 - 


Biographical Note 7 


Warren Montgomery was born on March 29, 1951 in Highland Park, Illinois. He 
grew up in Deerfield Illinois where he graduated from Deerfield high School in 1969. He 
received the Rensellaer Science Medal and placed 35th nationally in the M.A.A. high school 
mathematics competition. 


From 1969 to 1973, Warren attended Dartmouth College in Hanover New Hampshire. 
At Dartmouth, he worked part time on the operating system for the Dartmouth 
Time-Sharing System. He received the John G. Kemmeny prize in computing for his 
contributions to this system. He graduated summa cum laude, with a double major in 
Mathematics: and in Engineering. His thesis was entitled "Non-Adiabatic Behavior of 
Charged Particles in a Magnetic Null Sheet". 


From 1973 through 1978 Mr. Montgomery has been a graduate student at the 
Massachusetts Institute of Technology. He was supported for the first three years as a 
National Science Foundation Fellow. In 1976, he was awarded simultaneous degrees of 
Master of Science and Electrical Engineer. His S.M. and E.E. thesis was entitled "A Secure 
and Flexible Model of Process Initiation for a Computer Utility”. 


From 1976 to September of 1979 Mr. Montgomery was supported as a research 
assistant in the Computer Systems Research Group of the M.LT. Laboratory for 
Science. In the summer of 1977, he also worked for Prime Computer pple ead designing a 
communication protocol for a high-speed inter-processor communication networ 


Mr. Montgomery is a member of the Association for Computing Machinery, and its . 
Special interest groups on Operating Systems and Programming Languages. He is also a 
member of the Sigma Xi scientific honorary society. 


Mr Montgomery is currently working as a member of the technical staff of Bell 
Telephone Laboratories, investigating the design of software to support new home 
communication services. He is married to the former Carla P. Westlund. 


